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CHAPTER 1: INTRODUCTION: DETERMINANTS OF 


ONLINE ASSESSMENT ADOPTION 


This book is presented as a collection of four papers that have been prepared for publication. 
Each of the four papers presented in the book was written in the form of a journal article. The 
four papers prepared for submission as journal articles are presented in Chapter 2 through 5. 
Chapter 6 provides the summarized results, followed by a discussion of the implications for 
research and practice, and future directions for research that emerged from the findings. The 
current chapter provides a general introduction to the set of papers as a whole. The chapter 
first provides an overview of online assessment, and three popular technology acceptance 
models — the Technology Acceptance Model (TAM), the Technology Acceptance Model 2 
(TAM2), and the Unified Theory of Acceptance and Use of Technology (UTAUT). The 
chapter summarises their history, their reported efficacy, and some of their reported 
criticisms. The chapter concludes with the summarized aims of the overall research and 
explains the structure of the book. As a professionally-oriented doctoral work, the chapter 
commences with a personal statement on the professional experiences that motivated me to 


embark on the research. 
Personal Statement 


The research reported in this work was designed primarily to address a professional problem I 
encountered in my career. The inspiration for this research was derived from my own 
experiences, both as an elementary school teacher for over 17 years, and as an instructional 
designer for over 7 years in the education sector. As an elementary school teacher, I was 
passionate about enhancing the learning experiences of my students through educational 
technologies. As I ventured into my career as an instructional designer in the last 7 years, I 
have implemented numerous educational technologies in the institutions in which I have 
worked. Through my experiences, I have observed many failed implementations due to a lack 


of understanding of the importance of user acceptance in technology. For example, I was 
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engaged as an instructional specialist in Institute of Technical Education (Singapore) to 
design, build and implement a learning analytics system, Academic Intelligence System 
(AIS). The system was designed based on more from the senior management’s inputs and 
less from consultation from the actual users. The project was cancelled after the third year 
into the project by then incoming Chief Executive Officer of ITE, citing system mismatch 
and exorbitant development costs. Such observations have spurred me on to investigate ways 
to improve system implementations in these contexts by identifying the factors that influence 
technology acceptance. In my most recent years, I have served as a ‘functional head’, 
overseeing the implementation of online assessment in the ITE Examinations Division, and 
have had the opportunity to be involved the development, testing and implementation of an 
online assessment system. That experience prompted me to explore the acceptance of 
technology in the specific area of online assessment, which met with much resistance 


initially. 
Brief Introduction to Online Assessment 


In recent years, the delivery of educational assessments in many institutions has been shifted 
from traditional pen-and-paper methods to various forms of online assessment with the use of 
computer technology (Cavus, 2015; Diprose, 2013; Dube et al., 2009; Hebebci & Usta, 2015; 
Stone & Zheng, 2014). The term online assessment is often used interchangeably with the 
terms electronic assessment or e-assessment (Jordan, 2013), computer-assisted assessment 
(Bull & McKenna, 2003; Sim et al., 2004), computer-mediated assessment (Huot, 1996) and 
computer-based assessment (Fluck et al., 2009). The primary characteristic of all of these 
approaches is the use of some computerised technology to deliver assessment tasks (Bull, 
1999; Chalmers & McAusland, 2002). Typically, online assessment is used as the delivery 
mode for multiple-choice questions, online or electronic submission and computerised 
adaptive testing (Collares & Cecilio-Fernandes, 2019; Wang & Kingston, 2019). 
Assessments at any stage of the learning process (i.e., for formative, diagnostic or summative 
purposes) can be delivered in an online format. Within university settings, online assessment 
allows faculties to have large candidatures selecting answers to questions on a computer that 
is connected to an internet site that contains a database. Instant and detailed feedback may or 
may not be enabled, depending on the intent of the assessment (i.e., whether it is a formative 
or summative task). The increased efficiency of online assessment means that educational 


institutions can do more with less (Alruwais et al., 2018). As Gipps (2005) noted, enhanced 
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efficiency and the potential to enhance pedagogical processes are the main reasons for using 
online assessment. With automated marking and feedback, online assessment is viewed as 
efficient, fast and reliable, making it useful, particularly in cases where large numbers of 
students are being tested. In March 2020, 150 countries closed schools and educational 
institutions across the world due to the coronavirus (COVID-19) pandemic, impacting over 
80% of the world’s student population. Educational institutions around the world scurried to 
change courses and programmes from face-to-face to online delivery mode. This transition 
had a severe impact on assessment and evaluation processes (Batmang et al., 2021; Hebebci, 
2021; Hebebci, Bertiz, & Alan, 2020; Hu & Huang, 2022; Kibici, 2022; Kilincer, 2021; 
Paramitha et al., 2022; Paudel, 2021; Paudyal & Rana, 2021; Sahu, 2020; Unger & Meiran, 
2020; Xhelili et al., 2021). The COVID-19 situation thus accelerated the development and 
implementation of online assessment systems in many education institutions (OECD, 2020). 
In some countries, examinations were either postponed or cancelled; in others, they were 
replaced by continuous assessments or online assessment for final examinations. The topic of 
technology use in education, and in particular, online assessment, has since received much 


attention in education research and practice (United Nations, 2020). 
Computer-assisted Assessment and Online Assessment 
Computer-assisted assessment, by definition, is the use of computers for assessing student 


learning (Bull & McKenna, 2003). Various authors have differentiated amongst types of 


computer-assisted assessment, with one example shown in Figure 1. 


Online 
assessment 
(web-based) 


Networked 


Figure 1. Different Types of Computer-assisted Assessment [Adapted from Conole, G., & 
Warburton, B. (2005). A review of computer-assisted assessment. ALT-J, 13(1), 17-31.] 
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In broad terms, computer-assisted assessment is defined as the use of computers for assessing 
student learning and covers the whole process of assessment involving test marking, analysis 
and reporting (Chalmers & McAusland, 2002; Conole, & Warburton, 2005; Bull & 
McKenna, 2004). For example, Optical Mark Reading (OMR) and portfolio collection are 
considered to be forms of computer-assisted assessment. OMR, also known as “mark 
sensing”, remains one of the widely used computer-assisted assessment methods at present. 
OMR uses a computer to mark scripts composed initially on paper. It is a technique to sense 
the presence or absence of marks by recognizing the depth of darkness on an answer sheet, 
usually filled with a pencil or ballpoint pen (Deng 2008). Electronic portfolio collections, 
another form of com puter-assisted assessment, are the use of a computer to collect scripts or 
written work (McLoughlin, 2003) while computer-based assessment involves using a 
computer programme to mark answers that are entered directly into a computer (Fluck et al., 
2009). This form is characterized by the interaction between the student and computer during 
the assessment process (Charman & Elmes, 1998). In computer-based assessment, test 
delivery and feedback provision are performed through the use of a computer. This form can 
be subdivided into standalone applications that only require a single computer, applications 
that work on private computer networks, and those that are designed to be delivered across 


public networks such as web-based online assessment (Conole & Warburton, 2005). 
From Computer-based Assessment to Online Assessment 


In order to take a closer look at online assessment, it is essential to differentiate various 
related terms that have been used in association with this term in recently published works. 
To this end, a ‘desktop research’ study was initially carried out to gather data on the terms 
used in the last two decades (from 2001 to 2020). In this study, a search was first conducted 
using the Education Resources Information Center (ERIC) database for abstracts containing 
terms related to online assessment. Each search consisted of one and only one specific phrase. 
For example, the phrase containing only “online assessment” was searched, and the number 
of times this term had appeared in abstracts between 2001 and 2020 was recorded. From the 
search, “Online Assessment” appeared most frequently in all publications listed, followed by 
“Computer-assisted Testing”. Other frequently appearing terms included “Online Testing”, 
“Computer-assisted Assessment” and “Online Examination”. In Figure 2 and Figure 3, the 
search results from the ERIC database in 2020 are shown. “Online Assessment” remains the 


most frequently used term, followed by “Online Testing”, “Computer-assisted Testing”, 
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“Electronic Assessment” and “Computer-assisted Assessment”. Therefore, for discussion in 
this paper, the phrase “online assessment” will be used consistently to refer to assessments 


conducted either online or with the aid of a computer device. 
Number of Publications from 2001 to 2020 


Online Assessment Es 7645 
Computer-assisted Testing DET 3867 
Electronic Assessment NN 3410 
Online Testing 3291 
Computer-assisted Assessment NN 2959 
Online Examination EN 2061 
Computer-based Assessment MEN 1559 
Electronic Testing BN) 1304 
Computer-based Testing MEM 1150 
Electronic Examination MM 781 
Computerized Testing HEM 666 
Computer-assisted Examination ME 653 
Technology Enhanced Assessment [EMM 557 
E-assessment MM 445 
Computerzied Assessment MM 423 
Computer-based Examination Ml 265 
Technology Delivered Assessment [J 233 
Technology Enhanced Testing [J 193 
Computerized Examination ] 108 
Technology Delivered Testing 9 103 
Computer-aided Assessment | 89 
Technology Enhanced Examination | 88 
E-testing | 80 
Technology Delivered Examination | 50 
E-examination | 39 
Computer-aided Testing 5 


Computer-aided Examination 2 


Figure 2. ERIC Database search on assessment-related abstracts from 2001 to 2020 [Number 
of Publications from 2001 to 2020; data extracted on 8 September 2020 From ERIC 


Database; https://www.eric.ed.gov/] 
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Number of Publications in 2020 


Online Assessment Es 250 
Computer-assisted Testing DN 112 
Electronic Assessment 93 
Computer-assisted Assessment I 85 
Online Testing DE 34 
Online Examination EN 53 
Computer-based Assessment BN 46 
Computer-based Testing ME 29 
Electronic Examination EE 23 
Electronic Testing ME 21 
E-assessment MJ 16 
Technology Enhanced Assessment HJ 15 
Computer-assisted Examination HM 15 
Computerized Assessment HM 14 
Computerized Testing MM 12 
Technology Enhanced Testing Ml 10 
Technology Delivered Assessment [J 9 
Computer-based Examination BH 6 
Technology Delivered Testing | 3 
Technology Delivered Examination |] 3 
Technology Enhanced Examination | 3 
Computerized Examination | 2 
E-testing | 2 
Computer-aided Assessment | 2 
E-examination | 2 
Computer-aided Testing | 1 


Computer-aided Examination 0 


Figure 3. ERIC Database search on assessment-related abstracts in 2020 [Number of 
Publications in 2020; data extracted on 8 September 2020. From ERIC Database; 


https://www.eric.ed.gov/] 
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Development Phases in Online Assessment 


Historically, three main phases can be seen in the development of online assessment (Figure 
4). The first phase from the 1960s to 1990s involved the use of computers in assisting 
assessment. The second phase from the 1990s to 2000s saw the emergence of adaptive testing 
and the rise of the learning management systems. In the third phase that commenced in the 


2000s, the use of Web 2.0 tools, artificial intelligence and analytics was introduced. 


Phase 1: 1960 - 1990 


1960 
Programmed Logic for 
Automatic Teaching 
Operations (PLATO) 


1967 


interactive, Computer- 
Controlled, Information 


(Tec it} 
Phase 2: 1991 - 2000 


1992 
Tripartite Assessment 
Delivery System 
(TRIADS) 


1995 
DIAGNOSYS 
Question Mark for Web 


1997 
Blackboard 
ASSYST 


2000 
PILOT 


| Phase 3: 2001 - Present _ 2001 - Present 


2004 
Marktool 


2008 
Electronic Assessment 
System for Young 
Learner (EASY) 


1998 
ExamSoft * 


2010 


Penmarked 


2015 


OpenEssayst 


2018 
AuTomated assEssment 
Management System 

{ArTEMIS} 


Figure 4. Computer-based Examination Systems, Online Assessment Systems and 


Assessment Tools (1960 to Present) 
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Phase 1: Computer-assisted and Computer-based Assessment (1960 — 1990) 


In the period from 1960 to 1990 (Phase 1), computers were the conventional electronic means 
of delivering online assessment, and assessments or testing were primarily computer-assisted 
or computer-based. Examples of computer-assisted and computer-based tools used during 
this phase included databases, spreadsheets and expert systems. The first apparent attempt to 
use computers to assist the assessment process in the early 1960s was the Programmed Logic 
for Automatic Teaching Operations (PLATO) project, initiated by the University of Illinois 
(Smith & Sherwood, 1976, Woolley, 1994). The users of PLATO ranged from grade school 
students learning reading and mathematics, to graduate students learning complex concepts in 


the medical sciences. 


The other example of a large-scale assessment project that used computers during this period 
was the Time-Shared, Interactive, Computer-Controlled, Information Television (TICCIT) in 
1967 (Anderson, 1976). Other earlier attempts included the use of computers to automatically 
assess student programming assignments (Douce et al., 2005; Forsythe & Wirth, 1965; 
Hollingsworth, 1960). The assessment process was then affected by the revolution of 
microcomputers in the 1980s (Reiser, 2001). In the 1980s, there was an increased interest in 
using computers in instruction, and computers were used in automating instructional design 
tasks (Rottmann & Hudson, 1983). During the same period, large-scale multiple-choice tests 
were administered through the means of machine-readable forms, known as Optical Mark 
Recognition (OMR) forms, which are still in use today. In assessments facilitated by the use 
of OMR forms, students who are taking examinations shade their answers to selected- 
response questions on sheets that have been specifically designed for such a purpose (Jordan, 


2013). 
Phase 2: Adaptive Testing and Web-based Assessment (1990 — 2000) 


Phase 2 took place from 1990 to 2000, when computer capabilities increased to provide a 
broader range of options such as data processing and simulations. During the same period, 
many other sophisticated systems had emerged like the Tripartite Assessment Delivery 
System (TRIADS) from the University of Derby in 1992, which included varied question 
types to test higher-order skills (Allen, 1998; Boyle & Hutchison, 2009; Burrow et al., 2005; 
Cox et al., 2008). TRIADS is still in development today. 


| 


Determinants of Online Assessment Adoption in Singapore Technical College 


Another sophisticated system that emerged then was DIAGNOSYS, an adaptive testing 
software designed to prepare undergraduates for the study of physics and engineering 
(Appleby et al., 1997). Such adaptive testing systems measure users’ abilities by ‘building’, 
dynamically, an individualised test for each user (Chang & Ying, 1996). DIAGNOSYS, first 
developed in 1995, is based on a hierarchy of skills, in which each question delivered to a 
user depends upon on the previous answer given. The test items are selected sequentially, 
according to the current user’s performance. As a consequence, the test is tailored to each 
user’s ability by adjusting the difficulties of the items delivered to the responses given by the 
user. This means that higher-achieving users can avoid responding to a large number of easy 
items, and lower-achieving users are not confronted with a large number of items that are too 
difficult. During the same period that DIAGNOSYS was developed, there was the increased 
use of the World Wide Web, which led to the first commercial launch of web-based testing 
software, Question Mark for Web (Bull & Stephens, 1999). 


Towards the end of Phase 2, a decline in computer-assisted or computer-based approaches 
could be seen, with an increasing emphasis on web-based assessments. The exponential 
growth of internet usage seen at this time was a key factor in the decline observed. For 
instance, Blackboard, a widely used web platform released in 1997, had the capability to 
provide automatic grading of multiple choice and True/False questions. Systems such as 
QUIZIT (Tinoco et al., 1997), WebCT (Goldberg & Salari,1996), ASSYST (Jackson & 
Usher, 1997), ExamSoft® (Wadley et al., 2014) and PILOT (Bridgeman et al., 2000) were 
also examples of web-based systems with the ability to deliver and facilitate online testing 
and grading. These applications reduced emphasis on the more traditional forms of computer- 


assisted or computer-based approaches that had been developed previously. 
Phase 3: Web 2.0 Tools, Artificial Intelligence and Analytics (2000 — Present) 


Phase 3 began in 2001 and incorporates developments to the present day. During this period, 
e-learning took the forms of virtual classrooms, computer-mediated communications and 
online cooperative learning. Examples of online assessment tools that appeared over these 
years included the use of e-portfolios, blogging, social networking and web authoring 
systems (Conole & Alevizou, 2010). In 2005, WebCT was acquired by Blackboard, and the 
web-based system was retired in 2013 (Seepersaud, 2011). Over the last two decades, other 


prominent learning management systems with proprietary assessment capabilities were also 
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released. For example, the Moodle learning management system, an open-source virtual 
environment, had a significant influence on the development of online assessment tools 


(Jordan, 2013). 


Besides the growth in the number of learning management systems with online assessment 
features, many online assessment tools have also emerged in the market in recent years. 
These online assessment tools focused more on supporting teachers’ assessments and grading 
than on automated evaluation. For instance, MarkTool (Heinrich & Lawn, 2004) introduced 
an onscreen marking tool that allows markers to annotate PDF documents sent by students 
with formative feedback. The annotations are either textual and graphical and can be recorded 
and linked to each student. Penmarked (Plimmer, 2010), another software solution, supports 
the marking and annotating of students’ assignments with free-form ink annotations and 
associated marking tasks, like gathering and returning assignments, and recording grades. 

The last decade also ushered in systems that utilized more advanced techniques to enrich the 
assessment process, such as semantics, artificial intelligence, natural language processing, or 
personalised questionnaires. For instance, Hirata and Brueckner (2008) used AI devices such 
as neural networks, decision trees, and inference engines to support the creation of questions 
and to keep track of students’ learning. They proposed a framework for an Electronic 
Assessment System for Young learners (EASY). The approach generates results at the end of 
an assessment and permits revision, as well as a follow-up stage, by using the records of the 
answers. The specific methods used are: (1) clustering techniques, which enable the system to 
group the characteristics of the learning objects; (2) forward chaining, which is used to 
identify objects that users are thinking of when using the system; and (3) backward chaining, 


which checks the answers and allows for revisions based on these answers. 


In another example of an advanced assessment system, Jordan and Mitchell (2009) proposed 
a natural language-based system, Intelligent Assessment Technologies (IAT), deployed by 
the UK Open University, to create and mark short-answer, free-text assessment tasks. 
Evaluations of the answer matching process provided by this system has been demonstrated 
to have similar or higher accuracy to that of expert human markers. Students attempt the 
questions online and are given detailed feedback on incorrect and incomplete responses, and 
can repeat the task immediately to learn from the feedback provided. Also in the area of 
natured language-based systems, the Supportive Automated Feedback for Short Essay 
Answers (SAEeSEA) project uses a Natural Language Analytics engine to provide feedback 
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on students’ essays for summative assessment (Ras at al., 2015). OpenEssayist, which was 
part of the SAEeSEA, is a real-time learning analytics tool which provides automated 
feedback on a draft essay, operating through a linguistic analysis engine that is imbedded 


within a web-based application. 


In the context of assessing computer programming abilities, automated assessments for 
computer science courses were also developed such as the Kattis in 2002 (Basnet et al., 2018; 
Enstrom et al., 2011). Krusche and Seitz (2018) also introduced AuTomated assEssment 
Management System (ArTEMiS) that automatically assesses solutions to programming 
exercises and provides instant feedback so that students can iteratively solve the problems 
presented. ArTEMiS provides an online code editor with interactive exercise instructions. 
The system is programming language agnostic and applies to a variety of computer science 
courses. Using ArTEMiS, students gain experiences in version control, dependency 
management and continuous integration while attempting to solve programming tasks. 
ArTEMiS is suitable for beginners in programming, and helps students to realise their 
progress and gradually improve their programming solutions. In doing so, it can also reduce 
workloads for computer science instructors with large classes, and enhance students’ learning 


experiences. 


As we trace the development phases of online assessment, from the first use of computers to 
assist assessment to the current auto-marking web-based assessment, it is clear that 
technology has had a significant impact on assessment. As more and more educational 
institutions adopt online assessment, it raises questions on how teachers and students accept 
and use new assessment approaches. However, numerous questions have not yet been 
addressed on the subject of how these new approaches are being adopted in practice by 
educational institutions. According to Stédberg (2012), who examined a total of 76 articles 
published in three journals from 2006 to 2012, empirical studies on e-assessment had mainly 
focused on formative assessments by that time. Research approaches in this field have been 
highly disparate, and have typically focused on specific applications within particular 
contexts. As a result, it is often not possible to compare results across studies. Thus, in 
addition to an obvious need for further research on how teachers and students accept new 
technologies in assessment, there is also a need for a more systematic approach to the study 


of online assessment adoption in research. 
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Brief Introduction to Technology Acceptance 


The use of technology in education has been ever-increasing, and it has become an essential 
tool for a wide range of activities, from administrative tasks to teaching and learning. 
However, various authors have noted that the success of technology-related implementations 
will depend heavily on levels of user acceptance (Abu-Al-Aish & Love, 2013; Al-Adwan et 
al., 2013, Gibson et al., 2008; Ma et al., 2005, Salloum et al., 2019), and several technology 
theories and models related to how such user acceptance should be measured have now been 
developed. Studies of this kind are useful in assisting the design and implementation of 
systems based on an understanding of the factors that will affect users’ inclinations to apply 
the systems in their educational practices. In the context of the current work, this begs a focus 


on the intentions of lecturers to adopt online assessment systems within their courses. 


The Unified Theory of Acceptance and Use of Technology (UTAUT) (Venkatesh et al., 
2003) is the most recent widely applied and validated model which focuses on factors that 
influence users to adopt and use technology in various contexts. The UTAUT model was 
developed as a theoretical advancement over existing theories that were being adopted at the 
time to examine technology acceptance and usage. The authors reviewed, mapped and 
integrated constructs from eight earlier technology acceptance theories and models, and 
developed a unified model which incorporated common constructs across the eight theories. 
The following sections will examine prominent models that have appeared in the literature 
such as the Technology Acceptance Model (TAM) and TAM2, and how they eventually 
evolved into the UTAUT. 


Across all of these models, user acceptance of technology is operationalized in terms of either 
one or two ‘outcome’ variables. Consistently, the models incorporate Behavioural Intention 
(i.e., the respondent’s intention or otherwise to use the technology) as one index of user 
acceptance, and in many studies, this will be the only index that is incorporated. Some studies 
also, however, include an outcome variable called ‘Use Behaviour’ or alternatively, ‘Actual 
Use’ as another index of acceptance. These two variables measure either self-reported actual 
usage (e.g., as a frequency of use within a specified period) or can be based upon actual logs 


taken from the relevant system. 
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Technology Acceptance Model (TAM) 


TAM, first proposed by Davis in 1986, is a widely applied theoretical model designed to 
explain and predict user behaviour with respect to information technology (Legris et al., 
2003). The model is based on an adapted form of the Theory of Reasoned Action (TRA) put 
forward by Ajzen and Fishbein (1980). Davis (1986) adapted the TRA model by dropping the 
subjective norms and added perceived usefulness (PU) and perceived ease of use (PEOU) as 
belief variables. PU is the extent to which a user believes that using a specific system or 
technology enhances his or her job performance. PEOU is the extent to which a user believes 
that using a specific system or technology will involve physical and mental effort, to predict 
the user’s attitude towards a system (Figure 5). According to Davis’s (1986) initial proposal, 
a user’s overall attitude toward using a system was proposed to be a function of PU and 
PEOU, while PEOU had a significant effect on PU. However, Davis (1986) found that 
attitude did not fully mediate the effect of PU and PEOU on actual system use (David 1989). 
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Figure 5. Original Technology Acceptance Model [Adapted from Davis, F. D. (1986). A 
technology acceptance model for empirically testing new end-user information systems: 


Theory and results (Doctoral dissertation, Massachusetts Institute of Technology) | 


Davis, Bagozzi and Warshaw (1989) later modified the TAM model by adding attitude as a 
mediator between the belief constructs and intention to use (see Figure 6). The intent behind 
this modification was to increase TAM’s ability to predict users’ technology acceptance from 


their intentions, and its ability to explain users’ intentions in terms of their attitudes, 
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subjective norms, PU, PEOU, and related variables (Davis et al., 1989; Lai, 2017). In a 
longitudinal study with 107 users, the findings indicated that PU strongly influenced 
intentions and that PEOU had a small but significant effect on intentions as well, although the 
effect diminished over time. Attitude, on the other hand, partially mediated the effects of PU 
and PEOU on intentions, while subjective norms had no effect on intentions (Davis, 1989). 
Another modification by Davis et al. (1989) was the introduction of external variables that 
might influence the beliefs of a user towards a system. Examples of external variables are 
system characteristics, user training, user participation in system design and the nature of the 


implementation process. 


External 
Variables 


Figure 6. Modified Technology Acceptance Model [Adapted from Davis, F. D., Bagozzi, R. 
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P., & Warshaw, P. R. (1989). User acceptance of computer technology: a comparison of two 
theoretical models. Management Science, 35(8), 982-1003] 


However, in the final TAM model proposed by Venkatesh and Davis (1996), attitude was 
removed as a mediator based on findings that PU and PEOU had a direct influence on 
behavioural intention (Venkatesh & Davis, 1996). The final version of TAM is shown in 


Figure 7. 
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Figure 7. Final Technology Acceptance Model [Adapted from Venkatesh, V., & Davis, F. D. 
(1996). A model of the antecedents of perceived ease of use: Development and test. Decision 


Sciences, 27(3), 451-481.] 
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Views on the original TAM have varied, and limitations of the studies conducted to evaluate 
its efficacy have been highlighted. For example, Legris, Ingham and Collerette (2003) 
reviewed 22 selected articles published from 1980 to 2001 on TAM. While the researchers 
agreed that TAM was a useful theoretical model in explaining use behaviour in information 
system implementation, they underlined three limitations. First, nine of the studies reviewed 
involved students, though the intent was to evaluate user acceptance in industry. The 
researchers posed that results obtained with students may not generalise well to business 
environments (Lee et al., 2003; Yousafzai et al., 2007). Second, they noticed that most TAM 
studies examined the introduction of office automation software or system development 
applications, and posed that research would benefit from examining the introduction of 
business process applications. Third, they noted that most of the TAM studies did not 
measure actual system use, but rather, variance in self-reported use. Yousafzai et al. (2007) 
similarly pointed out in their meta-analysis of TAM findings that self-reported usage is 
subjective, and that actual usage or frequency recorded by the computerised system is a more 


useful measure of TAM’s efficacy in predicting behaviour. 


In contrast, various authors have affirmed TAM as a useful model for studying technology 
acceptance. For example, King and He (2006) conducted a meta-analysis of 88 TAM 
empirical studies from 1998 to 2003. They concluded that the TAM constructs of PU and BI 
were significant measures for technology acceptance and could be used in a variety of 
contexts. However, the authors also made a note of the fact that even when professionals’ 
acceptance of technology was the actual target of study, the empirical results of TAM studies 
were often based on students as surrogates for professionals (Legris et al., 2003). Yousafzai 
et al. (2007) argued that students may not provide a good representation of a ‘typical user’ 
owing to their age range, limited experience with technology, relatively low income, and 
different motivations for the use of technology. The fact that students’ use of technology is 
generally voluntary, unlike the use of technology in real work-place environments was cited 


as an additional limitation to the generality of results obtained in such studies. 


There have also been study findings which suggest that TAM studies conducted in different 
cultural contexts yield disparate results. For instance, Schepers and Wetzels (2007) compared 
TAM in Western and non-Western contexts and found that subjective norms had a more 
significant impact on behavioural intentions in Western studies. PU also seemed important in 


Western cultures, while PEOU had more relevance in non-Western studies. 
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In another meta-analysis study, Turner et al. (2010) examined 79 relevant empirical studies 
and found that most studies had small sample sizes. The authors also argued in this study that 
was a need to use an objective measure like actual usage that includes a computer-recorded 
usage or system logs. This echoed one of the concerns raised by Legris et al. (2003), who 
called attention to the fact that in many studies, self-reported usage had been assumed to 
reflect actual usage, though this assumption could be questioned. The researchers also 
recommended that TAM studies report more contextual information about the technology 


being evaluated and the populations being sampled in future studies. 


In their study on adopting TAM in mandatory environments, Hwang et al. (2016) pointed out 
that the parsimony of the model (i.e., its theoretical simplicity) has been both its strength and 
limitation. They argued that while its simplistic approach had attracted many researchers to 
adopt the model in their studies, TAM had added very little to the technology acceptance 
literature over the years. In other words, in the view of these authors, the TAM model offers 
limited knowledge and understanding of the acceptance phenomenon and _ technology 


implementation processes. 


In summary, among the many criticisms, TAM remains a popular and credible model for 
examining various technological implementation in education contexts (Granié & 
Marangunic¢, 2019; King & He; 2006). TAM’s constructs, PEOU and PU, have been shown 
to be prominent factors that predict technology acceptance. In particular, PU has been shown 
to be an important determinant of the adoption of various educational technologies. Despite 


this, there are various limitations of the model in the views of many authors within the field. 
Technology Acceptance Model 2 


Venkatesh and Davis (2000) extended the original TAM to include additional constructs like 
social influence (subjective norms, voluntariness, and image), cognitive instrumental 
processes (job relevance, output quality, and result demonstrability) and experience to 
explain PU and usage intentions (see Figure 8). The new model, the Technology Acceptance 
Model 2 (TAM2), retained all of the original TAM constructs (i.e. PU, PEOU, Intention to 
Use and Usage Behaviour), but extended the model by adding the following constructs: 

(i) subjective norms, which refer to a person's perception that people who are important 


to him/her think encourage the behaviour in question (Fishbein & Ajzen, 1975); 
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(11) image, which refers to the degree to which use of an innovation is perceived to 
enhance a person's status in one's social system (Moore & Benbasat, 1991); 

(111) job relevance, defined as a user’s perceptions of the degree to which the system 
applies to his or her job (Venkatesh & David, 2000); 

(iv) output quality, defined as an individual’s perception of how well the system performs 
relevant tasks (Davis et al., 1992); and 

(v) result demonstrability, which refers to the tangibility of the results of using the 
innovation (Moore & Benbasat, 1991) 


TAM2 also included two moderator variables (i.e., variables that change the strength or 
direction of the effects of other constructs in the model) - voluntariness and experience. 
Voluntariness is the extent to which potential system adopters perceive the adoption decision 
to be mandatory or optional (Venkatesh & Davis, 2000). Experience refers to the level of 
experience that users have in adopting the systems investigated. In the extended model, 
Venkatesh and Davis posited that the direct effect of subjective norms on usage intentions 


might subside over time with increased system experience. 


Technology Acceptance Model 


Result 
Demonstrability 


Figure 8. Technology Acceptance Model 2 [Adapted from Venkatesh, V., & Davis, F. D. 
(2000). A theoretical extension of the technology acceptance model: Four longitudinal field 
studies. Management Science, 46(2), 186-204. ] 
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While retaining TAM’s original relationships between PU, PEOU, intention to use and usage 
behaviour, TAM2 posits that subjective norms, image, job relevance, output quality and 
results demonstrability will also be direct determinants of PU. In TAM2, subjective norms 
are also posed to be a direct determinant of both image and intention to use, while experience 
moderates the influence of subjective norms on both PU and image. Voluntariness, however, 
is posed to moderate the influence of subjective norms on use intentions. Studies adopting 
TAM2 to explore user acceptance have shown that this model can account for between 40% 
and 60% of variance in users’ perceptions of the usefulness of a given system, and between 
34% and 52% of the variance in their usage intentions. Both social influence processes (i.e. 
subjective norm, voluntariness and image) and cognitive instrumental processes (i.e. job 
relevance, output quality, result demonstrability and perceived ease of use) have been found 


to significantly influence user acceptance levels (see Venkatesh & Davis, 2000). 


However, when Tang and Chen (2011) compared the technology acceptance models, they 
found that the explained variance of TAM2 was much lower than that of TAM. In TAM, 
intention to use explained 45% of the variance, while it was only 34% in TAM2. In short, the 
inclusion of subjective norms to the TAM did not appear to improve the model’s ability to 
explain a user’s intention to use technology. This conclusion was later supported by Rondan- 
Catalufia et al. (2015), who made a similar finding (i.e., that TAM2 did not offer a better 
explanation for technology acceptance when they compared different versions of technology 
acceptance models). Hwang et al. (2016) explained the relatively poor prediction achieved for 
the TAM2 in terms of the TAM’s origins in the TRA. Like TRA, TAM was developed to 
explain and predict user behaviour. The other observed mechanisms were not central to the 
theory, and added little to the prediction and explanation of user intention and behavior. Put 
simply, Hwang et al. argued that the influence of the added variables would be mediated by 
the main variables, and as a result, these additional variables did not improve overall 


prediction. 
Unified Theory of Acceptance and Use of Technology Model 


More recently, Venkatesh et al. (2003) consolidated the previous TAM theories (Davis, 1989, 
Taylor & Todd, 1995) and models (Ajzen, 1991; Compeau et al., 1999; Fishbein & Ajzen, 
1975; Moore & Benbasat, 1991; Thompson et al., 1991) and proposed the Unified Theory of 
Acceptance and Use of Technology (UTAUT) (see Figure 9). In the UTAUT, four constructs 
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play a significant role as direct determinants of user acceptance and use behaviour (UB): (1) 
performance expectancy (PE), (2) effort expectancy (EE), (3) social influence (SI); and (4) 
facilitating conditions (FC). As theorised, attitude toward using technology, self-efficacy and 


anxiety are not direct determinants of behavioural intention (BI). 
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Figure 9. Background Theories and Unified Theory of Acceptance and Use of Technology 


In the model, PE is the extent to which an individual believes that using the system will 
enable him or her to achieve gains in job performance, while EE is referred to as the degree 
of ease associated with the use of the system. SI is the extent to which an individual perceives 
that important others believe he or she should use the system, and FC is the extent to which 
an individual believes that there is an existing organisational and technical infrastructure to 
support the use of the system. In the UTAUT, PE, EE, and SI are all proposed to be 
predictors of BI, and via BI as a mediator, of UB. Conversely, FC is not theorized to operate 
via BI, but more directly on UB, unless other predictors in the model are not present. 
Specifically, Venkatesh et al. (2003) pointed out that if EE is not included as a predictor of 
BI, FC will act as a significant predictor of BI. However, in the presence of both PE and EE, 
FC will not be a significant predictor of BI. 


In addition to proposing the base UTAUT model described, Venkatesh et al. (2003) also 


proposed that various ‘moderating variables’ (i.e., variables that affect the direction or 


strength of the relationships between variables in the base model) may be important in 
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different contexts (see Figure 10). For example, the developers posed that effect of EE on BI 
may be moderated by both gender and age, such that this effect decreases with users’ 
experience in adopting given systems. They also posed that age and experience could 
moderate the relationship between FC and BI, such that as experience increases, the 
relationship between these elements may become more significant, especially in older groups. 
While the potential impact of such moderators are recognised here, these factors are not 
typically included in studies that are based on the UTAUT. In light of this fact, potential 


moderating influences were not explored in the present research. 
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Figure 10. Unified Theory of Acceptance and Use of Technology Model [Adapted from 
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of 
information technology: Toward a unified view. MIS Quarterly, 425-478. ] 


Summary of Technology Acceptance Research to Date 


This chapter traced the development of technology acceptance models from TAM and TAM2 
to the eventual UTAUT. Amongst the differences and commonalities, each of these models 
has made unique contributions to the literature on technology acceptance in the last three 
decades. Generally, TAM specifies general determinants of individual technology acceptance 
and therefore can be and has been used to explain or predict user behaviours across a wide 
range of learning technologies and user groups (Davis et al., 1989; Samaradiwakara & 
Gunawardena, 2014). TAM2 is an extension of TAM with new constructs. While the intent 
was to include subjective norms to the model, this was found to lower the model’s ability to 


explain intention to use (Tang & Chen, 2011). 
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Two of UTAUT constructs are similar to TAM constructs. PE can be mapped to PU, whereas 
EE can be mapped to PEOU. The other two constructs, SI and FC, are from the Theory of 
Planned Behaviour (Ajzen, 1991). When the predictive power of the UTAUT has been 
compared to TAM and TAM2, it has emerged as a better model for predicting users’ 
intentions to use technology, typically explaining around 70% of the variance in intentions, 
almost twice as much as TAM2 (Tang & Chen, 2011; Samaradiwakara & Gunawardena, 
2014). Momani (2020) referred to the UTAUT as the most comprehensive model designed 
simplistically to examine actual usage behaviour with a small number of constructs and 
moderating variables, and noted that this model could be modified and applied in studies of 


any new technology and information system. 


There have also been UTAUT studies conducted to examine whether findings would vary 
when conducted in different cultural contexts. The UTAUT model was tested in the United 
States and China, and it was found that culture played an essential role in affecting 
technology adoption (Venkatesh & Zhang, 2010). The role of SI varied across these two 
countries, while the effects of other factors were similar to previous studies. In another study, 
Im et al. (2011) also found that the effects of EE on BI and the effects of BI on UB were 
more modest in a Korean sample than had been reported in previous, US-based studies. 
Based on these results, the degree of relationship between variables in the UTAUT model 


may vary across different countries and cultures. 
Current Issues on Technology Acceptance Models 


One of the observations from the examination of the three models is the removal of attitude 
as a construct after the development of TAM2. Rondan-Catalufia et al. (2015) argued that 
when the attitude construct was removed from TAM2, the explained variance of the model 
dropped drastically. Interestingly, attitude was also absent as a construct in the later UTAUT. 
According to Yousafzai et al. (2007) in their meta-analysis of TAM, they pointed out that 
although attitude had been removed from later TAM models, research indicated it correlated 
strongly with usage behaviour, especially in environments where usage was mandatory. That 
raises a question on whether attitude should rightfully be restored to technology acceptance 


models. 
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Venkatesh et al. (2003) had suggested that later UTAUT research should focus on identifying 
constructs that could add to the prediction of BI and UB over and above what was already 
known and understood by the researchers. In a later review, Venkatesh et al. (2016) examined 
and synthesized the information system literature on UTAUT from September 2003 until 
December 2014. The researchers made a classification on the types of extensions that could 
be made to the UTAUT model to enhance its prediction in different contexts. These included: 

(i) the inclusion of new exogenous constructs (i.e., independent constructs, or those that 
are not dependent on other constructs in the model) and mechanisms; 

(11) the inclusion of new endogenous constructs (i.e., constructs that are dependent on 
other constructs in the model) and mechanisms; 

(111) the inclusion of new moderating constructs or mechanisms (i.e., factors that influence 
either the strength or valence of relationships between variables in the core model); 
and 

(iv) the inclusion of new outcome constructs (i.e., end-point endogenous variables) and 


mechanisms. 
Details of the Present Research Program 


UTAUT has been used in a wide variety of technology contexts and has been found a useful 
model in most of these contexts. For instance, Wedlock et al (2019) concluded that the 
UTAUT model and its instrument could be used in the educational research settings to test 
the relationships between antecedent and posterior constructs of technology usage, user 
attitudes, integration intentions, and post-adoptive behavior, when the researchers traced the 
evolution of education technologies. This argument was corroborated by an European 
validation study by Nistor et al. (2013) on UTAUT as an educational technology model with 
a large sample (n=4589). The large-scale validation concluded that the UTAUT questionnaire 
displayed adequate validity and reliability. A similar large-scale study (n=1723) was also 
conducted in Turkey using the UTAUT model in the educational technology context (Gogus 
et al., 2012). As such, the UTAUT model proves to be a suitable candidate for examining 


educational technologies like online assessment in this research study. 


There has been a lack of research studies using the UTAUT model to study online assessment 
adoption, the closest being the study by Tan (2013) on students’ adoptions and attitudes 


towards computerised placement tests. Online assessment has always been linked to online 
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learning and its delivery system, as many learning management systems have both 
assessment features and functions. This standalone study on online assessment adoption 
presents an opportunity for an in-depth examination of the factors that may affect its 


adoption. 


The research reported in this work is presented as a series of standalone papers, drawing 
primarily on survey data gathered from lecturers within the Institute for Technical Education 
(ITE) in Singapore. The overarching purpose of the research was to explore how a modified 
version of the UTUAT model could be used to study users’ acceptance of online assessment 
in a higher education setting. Specifically, the modified UTAUT developed here was applied 
to study lecturers’ acceptance of a given online assessment used within ITE in Singapore. 

Paper 1 is a conceptual paper which proposes an extended UTAUT model, incorporating 
additional constructs alongside those in the original model. As noted previously, while the 
original UTAUT has been widely adopted in many studies, this original model excludes the 
attitude construct incorporated in previous models such as the TAM. The extended model 
presented in this paper re-incorporates the attitude construct, based on empirical research that 
has affirmed its utility for predicting users’ intentions. Another two constructs, usability and 
learnability, were also introduced to enrich the extended UTAUT model. Previously 


unexamined relationships among the constructs are also proposed in the extended model. 


Paper 2 is also a conceptual paper, in which research on the acceptance of online assessment 
by teachers and students is discussed. Although there are various studies on online 
assessment acceptance, these have largely focused upon students’ acceptance levels. In this 
paper, it is argued that lecturers’ acceptance levels will be a key determinant of whether 
online assessment systems are eventually adopted. This area of study has not attracted the 
same level of attention as research on students’ attitudes, which provides a key rationale for 
the research program conducted here. The UTAUT is then presented as a model that has the 


potential to integrate research within the field of user acceptance in online assessment. 


Paper 3 is an empirically-based paper. In the research reported in this paper, an instrument to 
study lecturers’ online assessment acceptance was adapted from the original UTAUT 
questionnaire. Additional items for usability, learnability and attitude constructs were 


included in the questionnaire, and a bipolar response format used to measure responses. 
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Evidence of the validity of this questionnaire in terms of its content, response processes, and 
internal structure is presented within the paper. 

Paper 4 focuses on examining the determinants of online assessment acceptance in ITE 
lecturers using structural equation modelling. The study examined whether the original 
UTAUT constructs, PE, SI and FC had significant positive direct effects on BI and UB. It 
also examines whether the component of attitude, included in the extended UTAUT, had 
significant direct and indirect effects on both behavioural intention and usage behaviour. The 
relationships between the two additional constructs included (usability and learnability) and 
other elements of the model were also explored. Papers 1-4 are presented in Chapters 2-5 of 
this book, respectively. Chapter 6 then provides a general conclusion from the research as a 
whole and discusses potential future research studies, as well as practical implications, that 


emerged from the findings. 
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CHAPTER 2: AN EXTENDED UNIFIED THEORY OF 


ACCEPTANCE AND USE OF TECHNOLOGY MODEL FOR 
EDUCATION CONTEXTS 


In early 2020, education institutions around the world were faced with unprecedented 
circumstances, as schools and universities closed their doors to contain the spread of COVID- 
19 (UNESCO, 2020). Responses in higher education were diverse, ranging from approaches 
in which established courses were offered with minimal adaptation through online meeting 
platforms such as Zoom and Microsoft teams, to the complete redevelopment of course 
curricula, allowing these to be offered in fully online, self-directed format (Crawford et al., 
2020). The degree of challenge confronted by institutions in the shift from face-to-face to 
virtual or online delivery modes is likely to depend on many factors, including technological 
factors, e-learning system quality, and cultural factors, in addition to self-efficacy and trust 
factors (Almaiah et al., 2020). One variable that has been cited consistently as a possible 


impediment to the success of such initiatives is that of end-user acceptance. 


Even prior to the rapid developments seen in response to the COVID-19 crisis, numerous 
studies had documented the variable uptake of learning technologies in universities based on 
end-users’ acceptance levels (Al-Adwan et al., 2013; Fathema et al., 2015; Scherer et al., 
2019). In connection with this research, many theoretical models have been proposed and 
applied to examine technology acceptance and usage in education over the last few decades. 
These include the Theory of Reasoned Action (TRA), the Technology Acceptance Model 
(TAM), the Theory of Planned Behaviour, and the Model of Personal Computer Utilization 
(Ajzen 1991; Davis 1989; Davis et al., 1989; Fishbein & Ajzen, 1975; Thompson et al., 
1991). These models offer different accounts of the factors that influence technology 


acceptance, which include specific attributes of the technology and contextual factors. 


Within this group, one of the original user acceptance models was the TAM, proposed by 
Davis in 1986. TAM, which is based on the TRA, is designed to explain why a user accepts 
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or rejects information technology (Ajzen & Fishbein, 1980; Davis, 1989; Davis et al., 1989). 
Within the model, Perceived usefulness (PU) is the extent to which an individual believes 
that using a particular system or technology will enhance his or her job performance. 
Perceived ease of use (PEOU) is the extent to which an individual believes that using a 
particular system or technology will require physical and mental effort. According to the 
TAM, one’s actual use of a technology system will be directly or indirectly influenced by the 
PU and PEOU of the system, as well as by one’s behavioural intentions and attitudes. The 
TAM also proposes that external factors (e.g., system design) will affect intention and actual 


use through mediated (i.e., indirect) effects on PU and PEOU (Davis, 1989). 


In 2003, Venkatesh et al. (2003) then developed the Unified Theory of Acceptance and Use 
of Technology (UTAUT) by consolidating eight previous TAM theories (Davis, 1989, Taylor 
& Todd, 1995) and models (Ajzen, 1991; Compeau et al., 1999; Fishbein & Ajzen, 1975; 
Moore & Benbasat, 1991; Thompson et al., 1991). The UTAUT has since been used 
extensively by researchers to explain technology acceptance and use in a variety of contexts. 
Researchers have also analysed the strength and robustness of the UTAUT for predicting user 
behaviours, and have confirmed its strong explanatory power (Khechine et al., 2016). The 
current paper reviews research on the UTAUT model and proposes an extended UTAUT 
model which may enhance its power to predict technology acceptance in the context of 
educational technologies. The next sections will examine the UTAUT model and its 
applicability across various types of educational technology. The proposed extended UTAUT 


model is then described. 
Unified Theory of Acceptance and Use of Technology Model 


In the UTAUT, four constructs play a significant role as direct determinants of user 
acceptance and UB: (1) performance expectancy (PE), (2) effort expectancy (EE), (3) social 
influence (SI); and (4) facilitating conditions (FC). In the UTAUT, attitude toward using 
technology, self-efficacy and anxiety are not direct determinants of behavioural intentions 


(BI). A diagrammatic representation of the UTAUT model is shown in Figure 11. 


In the UTAUT, PE is the degree to which an individual believes that using a system will 
benefit him or her in terms of job performance. EE is the degree of ease with which users can 


adopt the system (Venkatesh et al., 2003). SI is the extent to which an individual perceives 
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that ‘important others’ consider that he or she should use the system (Venkatesh et al., 2003). 
FC is the extent to which an individual believes that there is an existing organisational and 
technical infrastructure to support the use of the system (Venkatesh et al., 2003). BI is the 


individual’s intention to use the technology. 


Behavioural 
Intention 


Performance 
Expectancy 


Use 
Behaviour 


Effort 
Expectancy 


Facilitating 


Conditions 


Figure 11. Unified Theory of Acceptance and Use of Technology [ Venkatesh, V., Morris, M. 
G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward 
a unified view. MIS Quarterly, 425-478. ] 


With respect to the importance of these factors for predicting BI and UB, PE, EE, and SI are 
all proposed to be predictors of BI, and via BI as a mediator, of UB. Conversely, FC is not 
theorized to operate via BI, but more directly on UB, unless other predictors in the model are 
not present. Specifically, Venkatesh et al. (2003) pointed out that if EE is not included as a 
predictor of BI, FC will act as a significant predictor of BI. However, in the presence of both 


PE and EE, FC will not be a significant predictor of BI. 
Empirical Research using the UTAUT 


Since its development, the UTAUT has been used in numerous empirical studies to explore 
relationships between situational factors and technology use behaviour. For example, Bouzif 
(2017) examined students’ continued intentions towards the use of a learning management 
system using the UTAUT model, while Al-Adwan et al. (2018) studied mobile learning 
adoption in higher education. In some of these studies, the UTAUT has been used in its 
original form (Bervell & Umar, 2017; Liao, Shim et al., 2004; Prasad et al., 2018). In others, 


it has been used as a part of the primary theoretical model or has been used alongside at least 
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one other theoretical model (termed a UTAUT ‘integration’ study by Venkatesh et al., 2016). 
An example of a UTAUT integration study was conducted by Chen and Hwang (2019). They 
examined self-regulation in terms of metacognition and motivation using the UTAUT model, 
to determine influences upon 312 Taiwanese college students’ behavioural intentions to 
continue online courses. In a different context, Yang et al. (2019) examined a model that 
integrated UTAUT and the Connected Classroom Climate (CCC) model, in a study with 289 
college students in China. Similar applications of the UTAUT model have been reported by 
other researchers such as Radovan and Kristl (2017) and Thongsri et al. (2018). 


Many of the studies using the UTAUT, however, have incorporated part of or the complete 
UTAUT as a baseline model, but have modified the model to incorporate additional 
mechanisms (termed a UTAUT ‘extension’ study by Venkatesh et al., 2016). Four types of 
UTAUT extensions have been proposed in this literature: (1) the inclusion of new exogenous 
constructs (i.e., independent constructs, or those that are not dependent on other constructs in 
the model) and mechanisms; (2) the inclusion of new endogenous constructs (i.e., constructs 
that are dependent on other constructs in the model) and mechanisms; (3) the inclusion of 
new moderating constructs (i.e., factors that influence either the strength or valence of 
relationships between variables in the core model) and mechanisms; and (4) the inclusion of 
new outcome constructs (i.e., end-point endogenous variables) and mechanisms. As an 
example of a UTAUT extension study, Al-Adwan et al. (2018) added the new factors of trust 
expectancy, self-management of learning and system functionality to study BI to use mobile 
learning. Table 1 summarises studies that have applied the UTAUT in education contexts, 


either in its original form or as part of an integration or extension study. 


Table 1. UTAUT in Educational Contexts 


Technology / Author(s) Constructs Additional Construct(s) 
Theoretical 

Framework 

Web-based Liao et al. (2004) PE; EE; SI; FC; - 

Learning Intention to Use the 


System; System Usage 


Yakubu & PE; EE; SI; FC; BI; - 
Dasuki (2018) Actual Use 
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Technology / Author(s) Constructs Additional Construct(s) 
Theoretical 
Framework 
Learning Bouznif (2017) PE; EE; Superior Satisfaction 
Management Influences; Continued 
System Usage Intention 
Bervell & Umar PE; EE; SI; FC; BI; - 
(2017) UB 
Blended Prasad et al. PE; EE; SI; FC; BI; - 
Learning (2018) User Behaviour 
e-Learning Salloum & PE; EE; SI; FC; BI - 
Shaalan (2018) 
Online Chen & Hwang _—~wPE;, EE; SJ; BI Metacognition; 
Learning (2019) Motivation 
Mobile learning Wang etal. PE; EE; SI; BItouse Perceived Playfulness; 
(2009) m-learning Self-management of 
Learning 
Iqbal & Qureshi _ SJ; FC; Intention to Ease of Use; Perceived 
(2012) Adopt M-learning Usefulness; Perceived; 


Playfulness 


Abu-Al-Aish & 
Love (2013) 


PE; EE; BI to Use M- 
learning 


Influence of Lecturers; 
Quality of Service; 
Personal Innovativeness 


Mtebe & 
Raisamo (2014) 


Al-Adwan et al. 
(2018). 


PE; EE; SI; FC; BI 


PE; EE; SI; BI 


Trust Expectancy; System 
Functionality; Self- 
management of Learning 


Ali & Arshad PE; EE; SI; FC; BI Learners’ Autonomy; 
(2018) Content Quality Design 
Alasmari & EE; SI; FC; BIto Use — Learning Expectancy; M- 
Zhang (2019) Mobile Learning Learning Technology 
Technology; UB of Characteristics; Self- 
Mobile Learning management of Learning 
Technology 
Desktop Video — Lakhal et al. PE; EE; FC; BI General SI; Peer SI; 
Conferencing (2013) Autonomy 
technology 
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Technology / Author(s) Constructs Additional Construct(s) 

Theoretical 

Framework 

Interactive Wong et al. PE; EE; SI; FC; BI - 

Whiteboards (2013) 

Software Wrycza et al. PE; EE; SI; FC; BI; Professional Training 

Engineering (2017) UB Diffusion; Model 

tools Interchange 

Video-based Kissi et al. PE; EE; SI; FC; BI Task-Technology Fit; 

Instruction (2018) Perceived Control Over 
Time; Learning-Family 
Conflict 

Wiki Yueh et al. PE; EE; SI; FC; BI; - 

(2015) Actual Use 

Computer- Lin & Lin (2019) PE; EE; SI; FC; BI; Social Network 

supported UB Awareness 

Collaborative 

Environment 

Open Mtebe & PE; EE; SI; FC; BIto - 

Educational Raisamo (2014) | Adopt and Use OER; 

Resources Actual Use of OER 

Online Oh & Yoon PE; EE; SI; FC; BI; Flow experience; Trust 

Information (2014) UB 

Services 


Based on the literature cited in Table 1, the UTAUT has been applied extensively in 
empirical research on technology use behaviours. This is likely to reflect the high predictive 
power of the model compared with alternatives in the field. In their evaluations of various 
technology acceptance models, Samaradiwakara and Gunawardena (2014) compared existing 
theories and models and concluded that the UTAUT had the highest explanatory power 
amongst available models for explaining users’ technology usage intentions. They concluded 
that this was because the determinants of BI and UB in the UTAUT incorporated eight earlier 
technology acceptance models. During the development of UTAUT, longitudinal data from 
entertainment, telecom services, banking, and public administration were used. The 
conceptualised UTAUT model was empirically tested using original data from the four 
organisations and then cross-validated using new data from another two organisations from 
the financial services and retail electronics industries. This careful approach to development 
ensured that the UTAUT integrated elements of different models that had appeared 


previously, enhancing its ability to predict user behaviours across a variety of contexts. 
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Empirical Results on the Predictions of the UTAUT Model 
PE as a Predictor of BI 


Venkatesh et al. (2003) posited that of the various predictors in the UTAUT, PE would have 
the strongest influence on BI, a proposition that has been borne out in many later empirical 
studies. Liao et al. (2004) adopted the UTAUT in its original form and found a significant 
positive influence of PE on the variable ‘Intention to Use the System’ in their study of student 
acceptance of web-based environments. Similarly, Prasad et al. (2018) found in their study on 
international students’ reactions to blended learning that PE had a significant influence on BI. 
PE has also been found to remain an influencing factor on BI in ‘UTAUT extension’ studies 


(i.e., when additional constructs are included within the model). 


For instance, in addition to all original UTAUT constructs, Oh and Yoon (2014) extended the 
UTAUT to include ‘flow experience’ (the overall sense that a person is acting with full 
consciousness) and trust mechanisms (the degree of confidence with which one can depend 
on others), and found that PE continued to have a significant positive influence on BI. In 
another study by Wrycza et al. (2017), the model was extended with professional training 
elements (i.e., the perceived flexibility and expressiveness of a training tool) and model 
interchange constructs (i.e., the capability of the tool to integrate seamlessly). PE continued 


to have a significant positive effect on BI, despite the inclusion of these additional constructs. 
EE as a Predictor of BI 


In contrast to the consistent results obtained for PE, the empirical results on EE as a predictor 
have been largely inconsistent. This has been true irrespective of whether the model tested is 
the original UTAUT or an extended form of the model. For instance, EE had a negative 
influence on BI in the study by Liao et al. (2004), but had a significant positive impact on BI 
in studies by Mtebe & Raisamo (2014) and Prasad at al. (2018). 


Disparate results have also been obtained in extended UTAUT studies, with EE found to have 
varying influences on BI across different contexts (Ali & Arshad, 2018; Lakhal et al., 2013; 
Kissi et al., 2018; Lin & Lin; 2019; Wrycza et al., 2017). These inconsistent results may be 
attributable to the variable types of technology explored across studies. For instance, in 


UTAUT studies on learning management systems, EE has often been found to have no 
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significant effect on BI (Ali & Arshad, 2018; Bouznif, 2017; Liao et al., 2004; Salloum & 
Shaalan, 2018). However, when the UTAUT model has been applied in mobile learning 
studies, EE has often been found to have a significant effect on BI (Abu-Al-Aish & Love, 
2013; Al-Adwan et al., 2018; Alasmari & Zhang, 2019; Almaiah et al., 2019; Mtebe & 
Raisamo, 2014; Wang et al, 2009). 


ST as a Predictor of BI 


The empirical results of SI as a predictor of BI have also not been entirely consistent across 
studies that have used the UTAUT model. In those that tested the original UTAUT, SI has 
been found to have a significant positive influence on BI (Liao et al., 2004; Prasad et al., 
2018; Yueh et al., 2015; Mtebe & Raisamo, 2014; Salloum & Shaalan, 2018). However, in 
other studies, such as that by Wong et al. (2013), SI did not have any significant influence on 
BI. Similar disparities have also appeared across studies of extended UTAUT models. For 
example, Wang et al. (2009) extended the UTAUT by adding the constructs of perceived 
playfulness and self-management of learning in a study of 330 Taiwanese participants from 
five organisations (i.e., Aerospace Industrial Development Corporation, IBM Taiwan, 
National Changhua University of Education, Chung Chou Institute of Technology and 
Yuanlin Community University) and found that SI had a positive effect on BI, while Iqbal 
and Qureshi (2012) did not in their study of an extended UTAUT model (which included ease 
of use, perceived playfulness and perceived usefulness) with 250 students from 10 


universities in Pakistan. 
FC as a Predictor of UB 


In many studies that appeared subsequent to the original UTAUT development studies, UB 
has often been omitted as a construct, and as a result, FC as a predictor of UB has also often 
been omitted. For instance, in the UTAUT study on interactive whiteboard acceptance by 
Wong et al. (2013), UB was omitted. For UTAUT studies that have included UB as a 
construct, however, findings have again been inconsistent. Some studies have reported that 
FC has significant influence on UB (Oh & Yoon, 2014; Prasad et al., 2018; Salloum & 
Shaalan, 2018) while various others have not (Alasmari & Zhang, 2019; Mtebe & Raisamo, 
2014; Yueh et al., 2015). Therefore, the role of FC in predicting UB is unclear. 
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Implications for the Application of UTAUT across Different Forms of Technology 


The UTAUT has been found to have a high level of general applicability. In other words, 
elements of this model have been found to be able significantly to predict user intentions and 
behaviours across a vast array of user groups, situations, and forms of technology. Venkatesh 
et al. (2016) summarised UTAUT research contexts according to (1) types of users (e.g. 
students, teachers, government employees and physicians); (2) types of technology (e.g. 
tablet PC, internet, web-based learning environment); (3) types of task (e.g. learning, 
research, social networking); (4) the timepoint at which users’ adoption decisions were 
studied (e.g. adoption, use or adoption and use); (5) types of organisations (e.g. educational 
institutions, academic societies, government organisations); (6) geographical locations; and 
(7) the types of relationships that were studied (e.g., only UTAUT main effects, or the 
influence of UTAUT moderating variables). 


The next sections summarise some of the research that has been conducted using the UTAUT 
across different forms of technology within education. These applications have indicated 
different relationships between the constructs within UTAUT depending on the form of 
technology that is being studied. Various studies have incorporated extensions to the UTAUT 
depending again on the type of technology under study, with these kinds of study being 


particularly prevalent in the mobile learning area. 
UTAUT and Web-based Learning Management Systems 


A web-based learning management system (LMS) is an online software application that 
presents and manages educational content and determines and evaluates educational objects 
(Forouzesh & Darvish, 2012). The UTAUT model has been utilised in various studies on the 
acceptance of web-based LMSs. For example, the UTAUT has been applied to study 
undergraduates’ acceptance of LMSs in different countries (Bounzif, 2017; Liao et al., 2004; 
Salloum & Shaalan, 2018; Yakubu & Dasuki, 2018). In Australia, Prasad et al. (2018) also 
used the UTAUT to study postgraduate students’ behavioural intentions in the use of blended 
learning programs. In Malaysia, Bervell and Umar (2017) identified new relationships among 
the UTAUT constructs in a study on LMS acceptance by tutors, while Chen and Hwang 
(2019) adopted an UTAUT model to examine how metacognition and motivation influenced 


students’ behavioural intentions to continue online LMS-based courses in a Taiwanese 
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college. All of these studies have indicated that elements of the UTAUT significantly 


predicted users’ intentions and behaviours in the LMS setting. 


Numerous studies that have explored the use of the UTAUT model to study user acceptance 
in web-based LMS settings have, however, suggested that EE has no significant influence on 
BI (Ali & Arshad, 2018; Bouznif, 2017; Liao et al., 2004; Salloum & Shaalan, 2018) and that 
similarly, SI has no significant influence on BI (Bervell & Umar, 2017; Bouznif, 2017; 
Prasad et al., 2018; Yakubu & Dasuki, 2018). These observations are consistent with those 
reported outside the LMS setting (Ali & Arshad, 2018; Lakhal et al., 2013; Lin & Lin; 2019; 
Wrycza et al., 2017). One possible explanation for this is that the main participants in the 
studies on web-based LMSs have been university students, who do not use the LMS 
voluntarily. When the use of the system is non-voluntary, it is intuitively reasonable that the 
effect of EE and SI would be minimal. There is also likely to be no effect on BI as the 
students are simply required to use the system, which would make their peers’ or instructors’ 
expectations to do so irrelevant. Thus, the results of UTAUT studies in the LMS context are 


tenable. 
UTAUT and Mobile Learning 


Mobile learning refers to learning mediated with handheld devices (Barzegar, 2016). As 
compared to UTAUT studies in other educational technologies, a higher proportion of studies 
in mobile learning have extended the model to include other variables and constructs. For 
instance, in the studies conducted by Wang et al. (2009) and Iqbal and Qureshi (2012), the 
construct of “perceived playfulness’ was incorporated in the UTAUT model when studying 
mobile learning adoption. Self-management is another variable that has been of considerable 
interest to researchers in the mobile learning area. For example, studies by Al-Adwan et al. 
(2018) at four Jordanian universities, Alasmari and Zhang (2019) at a Saudi Arabian higher 
education institution, and Wang et al. (2009) at five organisations (i.e. Aerospace Industrial 
Development Corporation, IBM Taiwan, National Changhua University of Education, Chung 
Chou Institute of Technology and Yuanlin Community University) all introduced self- 


management as an additional construct in their UTAUT-based studies. 


Other, more situation-specific extensions to the UTAUT have also been studied with respect 


to users’ acceptance of mobile learning systems. For instance, Iqbal and Qureshi (2012) 
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included ease of use while Abu-Al-Aish and Love (2013) included the additional constructs 
of influence of lecturers, quality of service, and personal innovativeness to examine the 
factors influencing the acceptance of mobile learning in the study of 250 university students. 
Ali and Arshad (2018) added learners’ autonomy and content quality as constructs to the 
original UTAUT when studying 386 students’ acceptance of mobile learning. Besides 
examining self-management, Al-Adwan et al. (2018) also added trust expectancy and system 
functionality, whereas Alasmari and Zhang (2019) extended the UTAUT model by adding 
Learning Expectancy and M-Learning Technology Characteristics in a study of 1203 users’ 


acceptance levels. 


Among the studies that have applied the UTAUT model to explore the acceptance of mobile 
learning, the findings have mostly been aligned with Venkatesh et al.’s (2003) predictions. 
Specifically, PE, EE and SI have been found to be significant determinants of mobile learning 
acceptance, across numerous contexts (Abu-Al-Aish & Love, 2013; Al-Adwan et al., 2018; 
Alasmari & Zhang, 2019; Chao, 2019; Wang et al., 2009). However, some important 
departures have been evident. Specifically, several studies have suggested that FC directly 
influences BI, which departs from the theorisation in the original UTAUT model (Ali & 
Arshad, 2018; Almaiah et al., 2019; Bervell & Umar, 2017; Lakhal et al., 2013; Radovan & 
Kristl, 2017). For mobile learning acceptance studies, however, the relationship between BI 


and UB has rarely been examined. 
UTAUT and Instructional Devices 


UTAUT has also consistently been extended to study educational technology tools (Lakhal et 
al., 2013). For example, Wrycza et al. (2017) introduced two constructs, professional training 
diffusion (i.e. the perceived flexibility and expressiveness of a training tool) and model 
interchange (i.e., the capability of the tool to integrate seamlessly) to the original UTAUT 
model, to examine the acceptance of software engineering tools within Information Systems 
Development courses. Kissi et al. (2018) extended the UTAUT model by introducing 
constructs such as Learning—Family Conflict (1.e. household chores or outdoors activities that 
impede and interfere on students’ attention to study at home), Perceived Control Over Time 
and Task-Fit Technology to investigate urban-rural high school students’ acceptance of 
video-based instruction in the flipped learning approach. Findings in these contexts have been 


similar to those in applications of the UTAUT model to explore users’ responses to web- 
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based LMSs. In other words, the findings typically showed that: (1) EE has no significant 
influence on BI (Kissi et al., 2018; Lakhal et al., 2013; Wrycza et al., 2017) and that (2) SI 
had no significant influence on BI (Kissi et al., 2018; Lakhal et al., 2013; Wong et al., 2013; 
Wrycza et al., 2017). 


UTAUT and Online Collaboration Tools 


Online educational services refer to any education-related information and services provided 
over the Internet. Mtebe and Raisamo (2014) adopted the UTAUT to examine barriers to 
instructors adopting and using open educational resources. Oh and Yoon (2014) added two 
additional constructs, ‘flow experience’ and ‘trust’ to the original UTAUT in their study on 
predicting the use of online information services. In both studies, EE was found to have a 
positive influence on BI. However, results for other relationships between elements of the 
model have been more variable. For instance, Mtebe and Raisamo (2014) found that FC, PE 


and SI did not have a significant effect on BI. 


From the literature, some general observations can be drawn from the UTAUT research 
findings on various forms of educational technology. The original UTAUT model posits that 
PE will be the strongest determinant of BI. However, in the UTAUT studies on online 
collaboration tools, PE was found to have no significant influence on BI. In other words, the 
belief that online collaboration tools may not necessarily improve user performance did not 
deter them from using the technology. Among the UTAUT studies on web-based LMSs, 
instructional devices and online collaboration tools, EE was not found to influence BI in 
several studies. That is, in these studies, even when educational technologies were not 
perceived to be easy to use, prospective users still intended to use them for the affordances 
that they might bring. SI also did not consistently influence BI across all studies. Based on 
the diverse findings reported here, it appears that while the UTAUT has been found to be a 
versatile predictive model across contexts, the power of the constructs in the model to predict 


users’ intentions can vary depending on the specific form of technology under study. 
The Proposed Extended UTAUT Model 


As noted previously, various researchers have proposed extensions to the original UTAUT 


model, incorporating additional constructs within specific settings. This has been particularly 
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apparent in research that has applied the UTAUT to study mobile learning applications. Here, 
we propose an extended UTAUT to study the factors that predict users’ adoption of 
technology within education contexts (see Figure 12). This extended model includes two 
additional exogenous constructs (usability and learnability), which are proposed as predictors 
of PE, FC and SI. The construct of attitude, which is included in earlier TAM models, has 


also been re-incorporated as a predictor within the extended UTAUT model proposed. 


aecce Behavioural 
Intention 


| taal 
[eS 


Conditions 
Figure 12. Extended UTAUT Model [ Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, 


F. D. (2003). User acceptance of information technology: Toward a unified view. MIS 
Quarterly, 425-478. ] 


Usability is referred to as the degree of ease of use, to achieve system objectives with 
effectiveness, efficiency and satisfaction (Bevan et al., 2015; Jokela et al., 2003; Shackel, 
2009). This new construct has been proposed because various studies outside the UTAUT 
literature have reported that factors of this kind have a significant influence on users’ 
acceptance of educational technologies (Holden & Rada, 2011; Juarez Collazo et al., 2014; 
Lah et al., 2020; Lin, 2013; Tsakonas & Papatheodorou, 2008). 


Learnability is defined as the ease and speed with which learners can become familiar with 
the the features and capabilities of a system, which will depend heavily on the quality of the 
system interfaces (Jeng, 2005; Nielsen, 1994). Although Zbick et al. (2015) had included 


usability and learnability in an earlier TAM model to study mobile learning adoption, the two 
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constructs have yet to be used ina UTAUT study. Learnability has been incorporated in light 
of findings outside the UTAUT literature that have reported that the ease with which systems 
can be learned can significantly influence user acceptance levels (Burney et al., 2017; Chiou 


et al., 2009; Jeng, 2005; Zbick et al., 2015). 


From previous studies, PE is posed to be the strongest predictor of BI in the original UTAUT 
model (Venkatesh et al., 2003). While attitude has been omitted in the initial UTAUT 
development, follow-up studies have shown significant relationships between the UTAUT 
constructs and attitude (Botero et al., 2018; El-Gayar et al., 2011; Jairak et al., 2009; 
Shuhaiber, 2015; Sumak et al., 2010; Yakubu & Dasuki, 2018). Nassuora (2012) also found 
in the study on students’ acceptance of mobile learning that FC had a positive influence on 
attitude towards behaviour. Past studies had also found that SI is positively related to attitude 


(Botero et al., 2018; Nassuora, 2012; Shuhaiber, 2015; Sumak et al., 2010). 


Attitude, which was adapted from the TRA, was included in an earlier user acceptance model 
— the TAM (Ajzen & Fishbein, 1980; Davis, 1986; Davis et al., 1989). The TAM explains 
that the individual’s attitude towards a given situation combines with subjective norms to 
shape behavioural intentions, which in turn influences the individual’s actual behaviour. 
While attitudes were included in earlier TAM studies, it was excluded as a construct in the 
development of UTAUT. Despite this, in the UTAUT studies on tablet PC adoption by El- 
Gayar and Moran (2006), Moran et al. (2010) and El-Gayar et al. (2011), attitude toward 
using technology was found to have a significant influence on BI. Attitude has also found to 
have a significant influence on BI in studies on mobile learning adoption (Jairak et al., 2009; 
Nassuora, 2012; Thomas et al., 2013) and acceptance of educational technology tools like 
virtual lecturing system, mobile-assisted language learning system and social learning 


platforms (Botero et al., 2018; Khechine, & Augier, 2019; Shuhaiber, 2015). 
Conclusion 


While the UTAUT has been found to be a reliable and robust model to study and explain 
technology acceptance and use across various educational contexts, Venkatesh et al. (2003) 
proposed that the UTAUT could be further developed and extended in future studies across 
contexts. The extended UTAUT model proposed in this paper incorporates three new 


constructs: usability, learnability and attitude, which are proposed potentially to enhance the 
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predictive power of the UTAUT. With the proposed model, not only it addresses both the 
affective and cognitive aspects of technology acceptance (Taherdoost, 2018), it further 
examines possible relationships between the various constructs not theorised in the original 


UTAUT model. 


Although the extended UTAUT model proposed here was designed primarily for application 
in educational contexts, the additional constructs incorporated might also be relevant in user 
acceptance studies in business and government institutions. Future research is clearly needed 
to validate the utility of the extended model, perhaps comparing this with the original to 
determine which of the two have the highest explanatory power across different contexts. The 
power of this and other extensions of the UTAUT to predict users’ acceptance could then be 
compared across different types of users, types of technology and contexts, to provide a range 
of different models that are suitable for use in specific contexts. Such research would provide 
researchers and practitioners with a range of possible models that can be selected based on 


needs and objectives in given situations. 
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CHAPTER 3: STUDENTS’ AND TEACHERS’ ACCEPTANCE 


OF ONLINE ASSESSMENT IN HIGHER EDUCATION: 
RECOMMENDATIONS FOR FURTHER RESEARCH 


Educational technologies have evolved over the years to become an integral part of teaching 
and learning processes in the higher education sector. The early 1960s marked the first 
attempt to use computers to assist education assessment processes (Woolley, 1994), with 
web-based online testing software then introduced in the 1990s (Bull & Stephens, 1999). In 
more recent times, with reduced teaching resources and increased student numbers, teachers 
across all levels of education have needed to do more with less by adopting technology 
(Donnelly, 2014; Nicol, 2007). This trend has given rise to the rapid growth of online 
learning and assessment approaches within the higher education sector. As a consequence, 
over the last two decades, online assessment has come to replace paper-based assessments in 


many colleges and universities (Boitshwarelo et al., 2017). 


The novel coronavirus (COVID-19) pandemic saw education systems around the world 
confront tremendous challenges due to the shutting down of schools and university campuses. 
In this context, such institutions were forced to find ways to continue teaching and learning 
activities without the physical attendance of staff members or students (UNESCO, 2020). 
Inevitably, many schools and universities turned to online learning platforms to address this 
need. The same challenges were seen in terms of implementing student assessments, and 
UNESCO (2020) listed a shift to online assessment as one of the five main strategies that 


countries had adopted to manage high-stakes assessments during the COVID-19 crisis. 


In the next few sections, we discuss different forms of online assessment, as well as the 
potential of online assessment to enhance processes and outcomes in education institutions. 
This includes a consideration of how students and teachers have been reported to respond to 
online assessment, based on the existing literature. We then propose the need for an 


integrated theoretical model to direct future research on users’ acceptance of online 
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assessment approaches. The paper focuses exclusively on the use of online assessment within 
higher education, given that the challenges confronted by end-users within schools are likely 


to differ from those of users based in colleges and universities. 
Forms and Potential Advantages of Online Assessment 


The term online assessment is often used interchangeably with the terms, electronic 
assessment or e-assessment (Jordan, 2013), computer-assisted assessment (Bull & McKenna, 
2003; Sim et al., 2004), computer-mediated assessment (Huot, 1996) and computer-based 
assessment (Fluck et al., 2009). While there are subtle differences in the way that these terms 
are used across publications, all refer to the use of some computerised technology to deliver 
assessment tasks (Bull, 1999). Various types of computer-assisted assessment have been 
described in the research literature, including: 

e Optical mark reading (OMR), also known as “mark sensing”, in which a computer is 
used to mark scripts composed initially on paper. This is a technique which senses the 
presence or absence of marks by recognising their depth of darkness on an answer 
sheet, filled with a pencil or ballpoint pen (Deng et al., 2008). 

e Online or e-portfolios, in which a computer is used to collect scripts or written work 
(McLoughlin, 2003). 

e Computer-based assessment, which involves a computer program marking answers, 
entered directly into digital form (Fluck et al., 2009). This approach can be subdivided 
into stand-alone applications that only require a single computer with applications that 
work on private networks, and those designed to be delivered across public networks 
such as web-based online assessment (Conole & Warburton, 2005). 

Online assessment approaches have been reported to offer several advantages over paper- 
based assessment. These include: 

(1) Providing immediate and anonymous feedback to students on assignments and 
assessments (Barkley, 2002; Llamas-Nistal et al., 2013; Spivey & McMillan, 2014; 
Ridgway et al., 2004). Immediate feedback rewards well-prepared students and 
encourages students who did not perform well to enhance their performance. Teachers 
also have more control over when feedback is given using this approach. For example, 
teachers can set feedback to be delivered after a specific time interval once questions 


are completed. Varying degrees of feedback such as test scores, test scores with correct 
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answers, or test scores with detailed solutions may be provided. 

(11) Making assessment more efficient, particularly in cases where large candidatures are 
assessed (Gipps, 2005). With online assessment, teachers can test students on a wide 
range of topics in one short test easily (Boitshwarelo et al., 2017; Brady, 2005). The 
ability to create, manage, and deploy online assessment means that a large part of the 
manual grading work can be automated. This not only reduces the instructional and 
administrative costs of teaching courses with large enrolments, but also indirectly 
affects the amount of learning that takes place in the course by lowering the costs of 
administering more frequent assessments. 

(iii) Reducing costs. Placing course material online can result in significant cost savings 
because paper, copying, and distribution expenses are all reduced or sometimes 
eliminated. Copying and delivery of assignments to classes with massive enrolments 
are often costly and inefficient (Barkley, 2002). Faculties wanting to reduce 
expenditures are likely to support the transition from paper assignments and 
assessments to online assessment. 

(iv) Increasing assessment reliability. Online assessment has been found to improve 
examination and testing reliability with machine marking, improved impartiality, and 
enhanced question styles that incorporate interactivity and multimedia (James et al., 
2002; Mora et al., 2012). 

(v)Moving examinations out of regular class time, allowing teachers to cover more 
content or the same content in more depth (Barkley, 2002; Barua, 1999). Online 
assessment also allows the offering of flexible testing times, delivery periods and 
frequent testing (Spivey & McMillan, 2014). 

(vi) Facilitating distance-learning courses. As online assessment only requires a computer 
device and internet connection, there is no need for students to be on campus. As such, 
online assessment has the potential to transform teaching and learning by removing the 


constraints of time, distance and space (Cirit, 2015; Lei & Gupta, 2010). 
Students’ Acceptance of Online Assessment 


Various studies have affirmed that testing format typically has little or no effect on actual 
academic performance (Anakwe, 2008; Bloom et al., 2018; Escudier et al., 2011; Spivey & 
McMillan, 2014; Wadley et al., 2014; Zandvliet & Farragher, 1997). Based on the results of 
this kind, students do not appear typically to be disadvantaged through the use of online 
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assessment approaches. Despite these findings and the potential advantages of the approach, 
the success and adoption of online assessment in education settings will inevitably hinge 
upon its acceptance by end-users - namely, the educators and students. As a result, we must 
look at how educators and students perceive and respond to this approach, as well as its 


impact on the teaching and learning process. 


The majority of student acceptance studies that have appeared with respect to online 
assessment have focused on its use in medical and/or health education (Bloom et al., 2018; 
Boevé et al., 2015; Deutsch et al., 2012; Jawaid et al., 2014; Lewis & Sewell, 2007; Usir & 
Ahamad, 2017; Wadley et al., 2014), with a smaller number examining its use in subject 
areas such as engineering (Riera Guasp et al., 2018), foreign language studies (Fageeh, 2015) 
and social science education (Hewson & Charlton, 2019). This section presents a brief review 


of studies that have appeared internationally across these discipline areas. 


Deutsch et al. (2012) conducted a study at Leipzig Medical School, in which all enrolled 
fourth-year medical students in one year took a web-based ‘mock’ examination. The majority 
of participants indicated that they felt confident in dealing with computers, with female 
students being significantly less convinced of their abilities in this regard. Students’ views of 
online assessment were also found to improve after completing the assessment, with respect 
to their overall attitudes towards online assessment; its perceived ease of use; the perceived 
objectivity of the assessment; and their acceptance of computer or web-based methods in the 
teaching and learning process. Differences in attitudes across male and female students 
seemed to be attributable to differences in their perceptions of computer self-efficacy. 
Initially, females’ reservations about technical problems affecting the accomplishment 
of online assessment were significantly higher than those of males. However, female 
students’ overall attitudes towards online assessment were found to shift positively after 
undergoing the assessment. Although different before the test, male and female students’ 


attitudes were found to be similar after the exposure to the web-based examination. 


Kumar et al. (2013) surveyed 126 first-year medical students on the use of an online portal 
for assessment in India. Results indicated that students felt comfortable in using the online 
assessment approach and that they had favourable attitudes toward the immediate feedback 
this approach afforded. Other comments from students indicated that they saw additional 


advantages to the approach, which included the reduced potential for errors, the potential for 
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to enhance the testing of knowledge depth, and the flexibility afforded by the approach (e.g., 


the ease with which respondents can deselect answers on re-considering their choices). 


Jawaid et al. (2014) conducted a study with 173 Dow University of Health Sciences 
postgraduate residents on their perceptions of online assessment and their preferences for 
paper-based or online assessment. Results indicated that while 23.6% of the residents were 
not entirely confident in using the approach before sitting the online assessment, 64.8% were 
either confident or extremely confident in undertaking online assessments after their initial 
experiences with it. A common problem (28.9%) encountered by students was logging in, 
which would typically be an issue that is easily addressed. In all, 61.8% rated online 
assessment as better overall than paper-based assessment after experiencing it for the first 


time. 


Several studies have been conducted to examine students’ responses to online assessment in 
other health science disciplines. For example, in 2013, the Campbell University College of 
Pharmacy and Health Sciences adopted the use of ExamSoft® assessment program in all 
required courses in the Doctor of Pharmacy program (Bloom et al., 2018). In a survey of 269 
students who completed the assessments, findings showed no significant differences in 
perceived comfort with the approach based on gender, age, or prior experience with online 
courses or assessment. Although the student population as a whole felt comfortable using 
ExamSoft®, there was evidence that specific sub-populations held different perceptions. 
Younger male students were found to be more likely to be comfortable with ExamSoft®, 
which was attributed to a higher level of prior familiarity with computers. Students using 
computers for daily routine tasks, like taking notes in the class were also more likely to report 
that the feedback provided after the online exam was useful for understanding their 
performance. The same group of students was also less likely than those who reported 
difficulties in using ExamSoft® to perceive a negative impact on their examination 


performance. 


Cirit (2015) conducted a study with pre-service teachers to access their perceptions towards 
paper-based, online and alternative assessments and to examine whether their attitudes 
changed toward the types of assessment after Web 2.0 tools were implemented. The analysis 
of the survey data with 155 participants showed a positive attitude towards the use of online 


assessment methods. In particular, participants felt that online assessment appealed to 
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different types of learners, and was helpful because teachers did not have to be in the same 
physical location as the students. The participants reported a highly positive attitude towards 
online assessment for English language and teaching skills. They felt that online assessment 
could provide authentic tools that other assessment methods could not in English 
methodology courses. Most importantly, these participants agreed they would like to use 
online assessment methods in their English courses when they graduated and become 


teachers. 


A few studies on attitudes toward online assessment have also appeared focusing on students 
in engineering, computer science, foreign language and social science education. In one study 
focusing on first-year engineering students, Riera Guasp et al. (2018) studied perceptions of 
online examinations in the context of blended assessment with 463 students from Universitat 
Politécnica de Valencia, Spain. In this study, Auto-scored Computer-Based Assessment 
(ACBA) was used as part of the blended assessment in the subject of physics for first-year 
Engineering degrees. Results showed that there was an overall positive perception of the 
ACBA tool, especially with respect to its ease of use, and its utility during the learning 
process, because the ACBA tests helped students prepare for their examinations. Despite this, 


students were critical of the rigidity of the automatic scoring process used. 


In the area of foreign language studies, Fageeh (2015) reported that students were willing to 
convert to web-based assessment activities in a study of students’ and faculty members’ 
attitudes towards online testing using the Blackboard learning management system. The 
survey was conducted with 400 students and 25 faculty members, at the Faculty of 
Languages & Translation, King Kalid University. Findings indicated that the students were 
confident in taking the online assessments, and enjoyed using Blackboard for this purpose. 


They reported perceiving this approach as useful for undertaking their language assessments. 


In contrast to the positive results reported above, in a study with 401 first-year psychology 
students at the University of Groningen in the Netherlands, Boevé et al. (2015) found that 
approximately 50% of the students they surveyed indicated a preference for paper-based 
multiple-choice examinations before taking their first computer-based examinations, with 
25% indicating no preference for the medium of assessment, and only 25% indicating a 
preference for computer-based assessment. After completing their first computer-based 


assessment, 16% remained positive, 43% of students felt more positive, 12% remained 
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negative, 14% felt more negative, and 15% remained indifferent towards computer-based 


examinations. 


It is clear from this review that while the majority of studies conducted thus far have 
suggested an overall positive response from students toward online assessment, findings from 
others have been more mixed. Such mixed responses accumulated from system usage 
problems faced by students as well as the negative and indifferent attitudes towards online 
assessment system, and the automatic scoring process within (Boevé et al., 2015; Jawaid et 
al., 2014; Riera Guasp et al., 2018). While the level of detail provided in these papers did not 
allow for the identification of the critical factors that moderated students’ responses, it is 
likely that the specific characteristics of the online assessment approaches or systems used in 


each study contributed to this variability. 
Teachers’ Acceptance of Online Assessment 


While various studies have appeared on students’ perceptions of, and attitudes towards, 
online assessment, studies on teachers’ responses to online assessment have been more 
scarce. Among these studies, Jamil et al. (2012) surveyed 314 teachers in Pakistan 
universities on their perceptions on computer-based and paper-based examinations. Results 
indicated that while the majority of the teachers disagreed with the statement that computer- 
based testing was the ‘worst’ tool for assessment, they did note the need for a ‘master plan’ to 
introduce computer-based examinations at the national level. Collectively, they also agreed 
that using computers minimised clerical mistakes and that computer-based examinations 
allowed them to assess more students in less time. In an online study with a random sample 
of 25 teachers from the English department in King Khalid University, Fageeh (2015) 
confirmed that most teachers were willing to convert to web-based assessment activities 
using the Blackboard learning management system. Findings showed that teachers had 
positive attitudes to apply e-testing technology in delivering formative and summative 
assessment online, believing that this technology was an assistive learning tool that was 
complementary to e-learning. However, when the effect of age on attitudes on online 
assessment was examined, there was a difference in attitudes towards online testing across 
different age groups of teachers. The findings did not indicate which age group was more 
positive. This finding, however, does suggest a potentially important moderator variable in 


educators’ attitudes to the introduction of online assessment methods. 
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Hamsatu et al. (2016) conducted a study with 30 teachers in a Nigerian higher institution and 
found that although these teachers agreed that online assessment was helpful, the potential 
benefits were not reflected in students’ assessment performances. Teachers commented in 
this study that the use of online assessment might provide too much opportunity for the 
students to check their answers, and thus, could encourage ‘laziness’ in the students. The 
teachers also expressed the view that the assessment process should not be over-dependent on 
technology. However, the majority affirmed that online assessment was time-saving and 


lessened the burden associated with examination processing. 


Bloom et al. (2018) reported that faculty members had difficulties with computers when an 
ExamSoft® assessment program was implemented at the Campbell University College of 
Pharmacy and Health Sciences. Of the 35 faculty members who responded to the survey, 
68% reported they had trouble at least once while creating an online examination, and 59% 
reported having had at least one problem during the administration of an examination. The 
faculty members did not perceive an impact on examination performance due to the adoption 
of ExamSoft®, and the analysis of examination grades indicated no significant performance 
differences across paper-based and ExamSoft® examinations. Despite the difficulties 
encountered, educators in this study saw the benefits of ExamSoft®, and indicated preferring 


it to paper-based examinations. 


Amante et al. (2019) conducted a survey with 130 teachers and 424 students from Public 
Universities and Polytechnic Institutes of Portugal on the factors influencing digital 
assessment. Findings showed that approximately 70% of the teachers felt that there was a 
lack of knowledge on the use of technology in the teaching subject area (68%), as well as 
how online activities were assessed (67%). Other difficulties cited were the extra time 
commitments needed by teachers to implement the approach (38%), the additional effort 
needed to learn on the part of teachers (35%) and fears related to potential technical problems 
(35%). The researchers concluded that teachers’ perceptions of digital assessment were not 
directly related to the teachers’ age groups. Similar results were reported by Rolim and Isaias 
(2018), in which 168 higher education teachers in Portugal were surveyed on their views 
about using e-assessment approaches. The majority reported considering digital assessment 
as a beneficial alternative to paper-based assessments, but also cited “insufficient knowledge” 
as a potentially influential factor in whether online assessment would be accepted by 


colleagues and students. 
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Although the studies reviewed revealed that teachers had responded well to the transition to 
online assessment, implementation issues and challenges were also uncovered. For instance, 
additional workload and efforts to learn were required to use online assessment systems 
effectively (Amante et al., 2019; Rolim & Isaias; 2018). Like in the studies on students’ 
acceptance, the trust of online assessment process also casted a shadow on teachers’ 
confidence in the online assessment system and its automatic grading processes (Bloom et al., 


2018; Hamsatu et al., 2016). 


Need for an Integrated Model to Direct Research on Users’ Acceptance of Online 


Assessment Approaches 


To ensure that the advantages of online assessment for higher education are fully realised, 
this approach must ultimately be accepted favourably by its end-users. Most studies on the 
acceptance of online assessment to date have focused on students, with research on teachers’ 
acceptance of online assessment being comparatively limited (Chien et al., 2014; Imtiaz & 
Maarop, 2014). This is surprising, given that it is the teachers who design, administer and 
deliver the online assessments, and thus, will be the primary decision-makers in terms of 
whether the approach is adopted, the extent to which it is used, and the success with which it 


is integrated into ongoing teaching and learning processes (Amante et al., 2019). 


Many disparate studies on the factors that can impact teachers’ intentions to use technology- 
based pedagogical and assessment tools have appeared within the literature (Amante et al., 
2019; Bloom et al., 2018; Fageeh, 2015; Hamsatu et al., 2016; Jamil et al., 2012). For 
example, various authors have suggested that administrators should implement online 
assessment systems that are user-centric in their designs, providing simple, clearly explained 
and consistent navigations that allow both teachers and students to navigate through 
assessment items with ease (Webb et al., 2013). This remains, however, an area in which 
much progress could still potentially be made. In the authors’ views, progress within this field 
has been hampered by the absence of an integrating theoretical framework. Many papers 
published thus far, therefore, have focused on a range of factors that may be quite 
idiosyncratic to their specific contexts. We argue that a more general, systematic approach, 


drawing upon sound theoretical frameworks, is needed for the field to move along. 


Among the many technology acceptance models that have appeared to date, the United 
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Theory of Acceptance and Use of Technology (UTAUT) is a sound candidate for addressing 
such a need. In developing the UTAUT, Venkatesh et al. (2003) consolidated various 
previous Technology Acceptance Model (TAM) theories (Davis, 1989, Taylor & Todd, 1995) 
and related models of behaviour (Ajzen, 1991; Compeau et al., 1999; Fishbein & Ajzen, 
1975; Moore & Benbasat, 1991; Thompson et al., 1991). In the UTAUT, four constructs play 
a significant role as direct determinants of user acceptance and use behaviour: (1) 
performance expectancy, or the degree to which an individual believes that using the system 
helps him or her to attain gains in his or her job performance; (2) effort expectancy, or the 
degree of ease with which the user can deploy the system; (3) social influence, or the extent 
to which an individual perceives that important others believe he or she should use the 
system; and (4) facilitating conditions, or the degree to which an individual believes that 
there is an existing organisational and technical infrastructure to support the use of the system 
(see Figure 13). In the original UTAUT, attitude toward using technology, self-efficacy and 
anxiety are not direct determinants of behavioural intentions to use technology, though the 
exclusion of these elements has been contested by other researchers (Dulle & Minishi- 
Majanja, 2011; El-Gayar & Moran, 2006; El-Gayar et al., 2011; Khechine & Augier, 2019; 
Moran et al., 2010). 


We propose that an extended model that draws upon the strong theoretical foundations of the 
original UTAUT model could provide a systematic basis for further research into teachers’ 
intentions to use online assessment methods. This proposed extended model is depicted in 


Figure 14. 


Performance 
Expectancy 
Effort 
Expectancy 
Social 
Influence 
Facilitating 
Conditions 


Figure 13. Unified Theory of Acceptance and Use of Technology [Adapted from Venkatesh, 
V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information 
technology: Toward a unified view. MIS Quarterly, 425-478. ] 
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Figure 14. Extended UTAUT Model [ Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, 


F. D. (2003). User acceptance of information technology: Toward a unified view. MIS 
Quarterly, 425-478. ] 


In the proposed extended model, attitude has been introduced as a construct. The rationale for 
including attitude into the proposed extended model is that many previous UTAUT extension 
studies have found that attitude significantly influences users’ behavioural intentions (Botero 
et al., 2018; El-Gayar and Moran, 2006; El-Gayar et al., 2011; Jairak et al., 2009; Khechine 
& Augier, 2019; Moran et al., 2010; Nassuora, 2012; Shuhaiber, 2015; Thomas et al., 2013). 
The proposed extended model also includes two new constructs: usability and learnability. 
Usability is the degree of ease with which users can adopt the system to achieve their 
objectives with effectiveness, efficiency and satisfaction (Bevan et al., 2015; Jokela et al., 
2003; Shackel, 2009). Learnability refers to the extent to which users can quickly become 
familiar with the application and make use of all relevant features and capabilities, which will 
depend heavily on the quality of the system interfaces (Jeng, 2005; Nielsen, 1994). These 
additional constructs have been incorporated into the extended model based on findings that 
both usability and learnability have a significant impact on user’s acceptance of technology in 


different contexts (Jeng, 2005; Joo et al., 2011; Zbick et al., 2015). 
Conclusions and Recommendations 


To ensure that the advantages of online assessment for higher education are fully realised, the 


implementation must be accepted favourably by its end-users. Most studies on the acceptance 
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of online assessment to date have focused on students, with research on teachers’ acceptance 
of online assessment being comparatively limited (Chien et al., 2014; Imtiaz & Maarop, 
2014). This can make it difficult for administrators who wish to introduce online assessment 


systems to determine how this can best be done. 


In this paper, we have argued that an extended UTAUT can be used to better integrate 
research into enhancing the adoption of online assessment within universities by teaching 
staff members. In the extended UTUAT, users’ responses to technology still depend on 
factors such as performance expectancy (PE), facilitating conditions (FC) and social 
influence (SI), as in the original UTAUT. However, users’ attitudes, and the perceived 
usability and learnability of systems are also proposed to have a significant impact on users’ 
intentions to adopt the system (BI). The extended UTAUT is proposed here as a model that 
may increase the power with which teaching staff's acceptance of online assessment systems 


can be predicted. 


Moving beyond questions of design, the context in which online assessment approaches are 
introduced is also likely to be an essential factor in users’ acceptance of the approaches. The 
success of any shift to online assessment will require ‘buy-in’ from both the students and the 
teachers. The level of buy-in seen from teachers is likely to be a product of myriad factors, 
including the nature of the technology, the organisational context, and the model used to 
manage the change process (Legris et al., 2003; Orlikowski & Hoffman, 1997). Hence, 
faculties need to consider carefully how they should enact the change process when 
introducing new online assessment systems. The following strategies may help to ensure that 
teachers respond more favourably to shifts from paper-based to online assessment systems. 

(i) Ensure that proper scaffolding is applied in introducing teachers to the online 
assessment systems. This strategy will be important regardless of how useable and 
learnable a system is. Yuen and Ma (2008) commented that this step is crucial to build 
up teachers’ confidence in using technology in general. Increased confidence will, in 
turn, increase willingness to use other forms of instructional technology in the future. 
Intuitively, more straight-forward user interfaces are likely to appeal to teachers with 
little prior experience in online assessment systems. However, with increased 
experience, these teachers may also be willing to use more sophisticated systems. 

(11) Provide effective ongoing professional development to assist teachers in the adoption 


of online assessment tools. Studies have shown that the most effective professional 
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development programmes that improve teaching practices are those with activities that 
are ongoing and sustained over time (Tournaki et al., 2011). Besides providing 
professional development programmes, ongoing institutional support provision will 
also be essential. Buchan and Swann (2007) suggested a three-level support approach 
which includes real-time training and professional development, helpdesk 
troubleshooting, and self-help resources. High-quality technical support structures for 
different staff groups must also be readily available and accessible throughout the 
assessment periods. 

(111) Ensure that the additional initial learning requirements are factored into teachers’ 
workloads. The extent to which this must be considered will vary in part with how well 
the system is designed (i.e., systems that are more ‘learnable’ will entail a lower 
workload commitment than those less so). However, this factor is likely to be a 
significant moderator of responses to the introduction of online assessment or any other 
technology-based tool. In the study by Amante et al. (2019), teachers cited reasons such 
as “extra time spent by teachers” and “additional effort to learn by teachers” for not 
using online assessment tools. Results of this kind underscore the importance of 
considering the impact on workload as a potential barrier to the introduction of online 
assessment systems. 

(iv) Offer different levels of induction that can be tailored to the needs of individuals. As 
cited previously, some studies have highlighted significant relationships between age, 
experience and teachers’ attitudes towards online assessment. These results suggest that 
different levels of support in the introduction of online assessment may be needed for 
different teacher cohorts. Again, the extent to which this factor needs to be considered 


may be a product of the learnability of the system. 


Universities need to manage the paradigm shift from paper-based to online assessment well 
for such implementations to succeed (Amante et al., 2019). At present, however, only a 
limited body of evidence exists to guide higher education institutions on the best approaches 
to achieve this goal, and in particular, on how teachers are likely to respond to these shifts. 
Further research into teachers’ responses to online assessment, and into ways to overcome 
perceived barriers to its use in real settings, will be critical to ensure that its benefits are seen 
in improved teaching and learning outcomes. The extended UTAT model proposed here may 
assist in integrating research within this important field, to ensure that the benefits of online 


assessment methods are harnessed fully within the higher education sector. 
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CHAPTER 4: DEVELOPMENT AND VALIDATION OF AN 


INSTRUMENT TO MEASURE ONLINE ASSESSMENT 
ACCEPTANCE IN HIGHER EDUCATION 


With technological advances, assessment processes have shifted increasingly from traditional 
pen-and-paper methods to online forms in many education contexts (Cavus, 2015; Diprose, 
2013; Dube et al., 2009; Stone & Zheng, 2014). The term online assessment is often used 
interchangeably with terms such as electronic assessment (Jordan, 2013), computer-assisted 
assessment (Bull & McKenna, 2003; Sim et al., 2004), computer-mediated assessment (Huot, 
1996) and computer-based assessment (Fluck et al., 2009). These terms refer to the use of 
information technology to present, deliver, administer, or score assessment tasks (Bull, 1999; 
Chalmers & McAusland, 2002). Online assessment can range from a relatively minimal use 
of technology, such as the online or electronic submission of assignments, to more involved 
usages such as the delivery and scoring of multiple-choice questions and computerised 


adaptive testing (Collares & Cecilio-Fernandes, 2019; Wang & Kingston, 2019). 


Various potential benefits of online assessment have been cited by authors in the field, such 
as increased efficiency and accuracy of marking (e.g., Barkley, 2002; Boitshwarelo et al., 
2017; Brady, 2005; Cirit, 2015; Gipps, 2005; James et al., 2002; Lei & Gupta, 2010; Mora et 
al., 2012; Llamas-Nistal et al., 2013; Ridgway et al., 2004; Spivey & McMillan, 2014). 
Despite this, studies that have explored students’ and teachers’ acceptance of this approach 
have produced mixed results (Amante et al., 2019; Bloom et al., 2018; Boevé et al., 2015; 
Cirit, 2015; Deutsch et al., 2012; Fageeh, 2015; Hamsatu et al., 2016; Jamil et al., 2012; 
Jawaid et al., 2014; Kumar et al., 2013; Riera Guasp et al., 2018). 


One of the factors that has been cited as a possible explanation for these mixed results is that 
teachers may be reticent to adopt online forms of assessment, and as a result, may use these 
approaches in a minimal or suboptimal way (Amante et al., 2019; Bloom et al., 2018; Fageeh, 


2015; Hamsatu et al., 2016; Jamil et al., 2012; Rolim & Isaias, 2018). This literature relates to 
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teachers’ acceptance of online assessment, which is often operationalised in terms of their 
intentions to make use of the technology and, in some cases, their actual or reported use of 


that technology (Chien et al., 2014; Imtiaz & Maarop, 2014). 


Various studies on factors that can impact teachers’ intentions to use technology-based 
pedagogical and assessment tools have appeared within the literature (AlDahdouh, 2019; 
Barnett, 2018; Boz & Adnan, 2017; Bradley, 2021; Kaleli, 2021; Khashaba, 2020; Kibici & 
Sarikaya, 2021; Or & Chapman, 2021; Schneberger et al., 2008; Suryaratri, Komalasari, & 
Medellu, 2022; Terzis & Economides, 2011; Terzis et al., 2013a; Terzis et al., 2013b; 
Thompson & McDowell, 2019; Wiggins & van der Hoff, 2021). For example, several authors 
have suggested that to enhance educators’ acceptance of online assessment methods, 
administrators should implement systems that are user-centric in their designs, providing 
simple, clearly explained and consistent navigations, which allow both teachers and students 
to navigate through the items in the assessment with ease (Webb et al., 2013). The efficacy of 
these approaches for enhancing educators’ acceptance levels, however, has not yet been 


affirmed conclusively. 


The subject of educators’ acceptance of online assessment remains one in which much 
progress could potentially be made. In the authors’ views, progress within the area has been 
hampered by the absence of an integrating theoretical framework. Many papers published 
thus far, therefore, have focused on a range of factors that may be quite idiosyncratic to their 
specific contexts. We argue that a more general, systematic approach, drawing upon sound 
theoretical frameworks, is needed for the field to progress. We propose here that the Unified 
Theory of Acceptance and Use of Technology (UTAUT) is a suitable candidate for 


addressing this need. 


Samaradiwakara and Gunawardena (2014) reported, in their comparison of various existing 
theories and models in technology adoption, that the UTAUT had the highest explanatory 
power of all technology behavioural intention models that had appeared. The models against 
which the UTAUT was compared in that study included the Theory of Reasoned Action 
(Fishbein & Ajzen 1975); the Technology Acceptance Model (TAM - Davis, 1989); the 
Technology Acceptance Model — 2 (TAM2 - Venkatesh & Davis, 2000); the Motivation 
Model (Davis et al., 1992); the Decomposed Theory of Planned Behavior (Taylor & Todd, 
1995a); the Combined Technology Acceptance Model and Theory of Planned Behavior 
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(Taylor & Todd, 1995b); the Model of PC Utilization (Thompson et al., 1991); the 
Innovation Diffusion Theory (Rogers, 2010; Rogers and Shoemaker, 1971); and Social 
Cognitive Theory (Festinger, 1957). 


The Unified Theory of Acceptance and Use of Technology 


Venkatesh et al. (2003) consolidated previous Technology Acceptance Model (TAM) 
theories (Davis, 1989; Taylor & Todd, 1995a) and models (Ajzen, 1991; Compeau et al., 
1999; Fishbein & Ajzen, 1975; Moore & Benbasat, 1991; Thompson et al., 1991) to propose 
the UTAUT (see Figure 15). The UTAUT has been widely regarded since its inception as a 
model that synthesises past technology theories. The UTAUT was first tested empirically 
using data from four organisations, then cross-validated using new data from another two 
organisations from the financial services and retail electronics industries. Venkatesh et al. 
(2003) suggested that while the UTAUT was initially designed for research use in business 
contexts, the model could be further modified or extended to incorporate alternative or 


additional measures of users’ behavioural intentions and usage in education contexts. 


Performance Behavioural Use 
Expectancy Intention Behaviour 
Effort 
Expectancy 
Social 
Influence 
Facilitating 
Conditions 
Figure 15. Unified Theory of Acceptance and Use of Technology Model [Venkatesh, V., 


Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information 
technology: Toward a unified view. MIS quarterly, 425-478. ] 


The original UTAUT posited that behavioural intention (BI) and actual behaviour use (BU) 
associated with a given technology would be influenced by four main factors: performance 
expectancy (PE), effort expectancy (EE), social influence (SI) and facilitating conditions 


(FC). PE is the extent to which a user believes that using a system will help him or her to 
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achieve gains in job performance. EE is the degree of ease with which a system can be used, 
while SI is the extent to which a user perceives that ‘important others’ believe that he or she 
should use the system. Finally, FC is the extent to which an individual believes that there are 
existing organisational and technical infrastructures to support the use of the system. The 
authors also acknowledged that the influence of the four primary factors might be moderated 
by factors such as gender, age, experience and voluntariness. These ‘moderating factors’, 
however, are not part of the primary model, and are typically not included in empirical 


studies on the UTAUT. 
The Extended UTAUT Model 


An extended UTAUT model has been proposed by first author as depicted in Figure 16. 
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Figure 16. Extended Unified Theory of Acceptance and Use of Technology Model 
[ Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of 


information technology: Toward a unified view. MIS quarterly, 425-478. ] 


In this model, additional exogenous constructs, usability and learnability, have been 
incorporated as potential influences on both BI and UB. In the extended UTAUT model, PE, 
SI and FC have been retained as these have all been shown to be significant predictors of 
behavioural intention and usage behaviour. EE has not been included in the extended model, 


as many UTAUT studies have shown that its influence has not been as consistent that of the 
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other original constructs (Ali & Arshad, 2018; Alshehri et al., 2019; Bouznif, 2017; Chao, 
2019; Kissi et al., 2018; Lakhal et al., 2013; Lin & Lin, 2019; Salloum & Shaalan, 2018; 
Thongsri et al., 2018; Wrycza et al., 2017). Finally, attitude as a construct has been added, as 
this was first included in an earlier TAM model and reintroduced later in many extended 
UTAUT studies. Attitude has since been demonstrated to be a significant predictor of BI in 
numerous empirical evaluations (Botero et al., 2018; Dulle & Minishi-Majanja, 2011; El- 
Gayar & Moran, 2006; El-Gayar et al., 2011; Khechine & Augier, 2019; Moran et al., 2010; 
Nassuora, 2012; Shuhaiber, 2015; Sumak et al., 2010; Thomas et al., 2013). All exogenous 
constructs in the extended UTAUT model are posited potentially to have an influence on 


attitude, BI and UB. 
The Present Study 


Many instruments have been used to measure various aspects of the original UTAUT in 
previous empirical studies (El-Gayar & Moran; 2006, Raman et al., 2014; Sumak et al., 2010; 
Sumak & Sorgo, 2016; Yueh et al., 2015). However, no validated instrument has yet been 
developed which incorporates the additional constructs of attitude, usability and learnability 
within the extended model proposed by the first author. The aim of this study, therefore, was 
to develop and validate an instrument suitable for assessing educators’ acceptance of online 


assessment based on this extended UTAUT model. 


The study was conducted within one Institute of Technical Education (ITE) in Singapore, and 
the instrument developed referred specifically to the online assessment system used within 
that institution (the Integrated Assessment System, or IAS). ITE is a vocational institute in 
Singapore that provides pre-employment training to secondary school graduates and 
continuing education and training to working adults. The institution offers trade-related 
certifications and vocational diplomas for skilled technicians and workers in supporting roles 
in professions like engineering, accountancy, business administration, nursing, medicine, 
architecture, and law. IAS, which the first author was an administrator and system tester 
before its launch, has capabilities across areas of online assessment: authoring, delivery and 
marking and is supported with other features like analysis and reporting. The assessment 
system delivers online tests throughout the academic terms and online examinations 


quarterly. 
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An initial list of 34 items was first compiled based on an extensive search of literature that 
reported on related instruments that had been developed and published previously. This was 
later reduced to a final list of 20 items based on data collected from a pilot group of 


participants. The final questionnaire is shown in Table 2. 


Table 2. Extended UTAUT Questionnaire (20 Items) 


Construct Statement 
Performance Using IAS slows me downin © Using IAS enables me to 
Expectancy accomplishing tasks accomplish tasks quickly 
(PE) , <? ’ 
Using IAS decreases my Using IAS increases my 
(Venkatesh et sf oss ‘6 io ; 
roductivity terri roductivity greatly 
al., 2003) : - : 
My supervisor thinks that I €> My superior thinks that I 
should not use IAS should use IAS 
Social Influence 
(sp My colleagues think that I €> My colleagues think that I 
should not use IAS should use IAS 
(Venkatesh et = —— 
1. 2003) In general, the organisationhas “© In general, the organisation 
al. 
been unsupportive in the use of has been supportive in the 
IAS use of IAS 
I do not have any resource €> [have all the resources 
available to use IAS available to use IAS 
Facilitating IAS is incompatible with € _ IAS is compatible with all 
Conditions (FC) systems I use other systems I use 
(Venkatesh et <> A specific person (or 
No specific person (or group) ; ; : 
al., 2003) group) is easily and readily 
is available for assistance with 
available for assistance 
IAS difficulties 
with IAS difficulties 
I think IAS is extremely €> I think IAS is extremely 
difficult to use easy to use 
Usability © J think the various 
I think the various functions in - 
(Brooke, 1996) ; functions in IAS are well- 
IAS are not integrated ; 
integrated 
I think there are too many €% I think there is overall 
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Construct Statement 
inconsistencies in IAS consistency in IAS 
I need the support of a <2 
; I am able to use IAS on my 
technical person to be able to ; 
. own without any help 
Learnability use IAS 


(Brooke, 1996) 


I need to learn a lot of things C7 


before I could get going with 
IAS 


I need not learn new things 
before I could get going 
with IAS 


Behaviourial Given a choice, in the next 6 months: 
Intention (BI) I do not intend to use IAS €7 — [intend to always use IAS 
(Venkatesh et €> 

I do not plan to use IAS I plan to always use IAS 
al., 2003) 

All things considered, using IAS is: 
Attitude Bad ©? Good 
(Davis, 1989) Unfavourable €7% Favourable 

Negative <7 Positive 
Use Behaviour I seldom use IAS €7% — Talways use IAS 
(UB) €> 

i. I spend great amount of 
(Venkatesh et I spend little time on IAS 
time on IAS 
al., 2003) 
Method 


Instrument Development 


The original UTAUT questionnaire published by Ventakesh et al. (2003) was used as a basis 
for developing the extended UTAUT questionnaire. Various revisions, however, were 
incorporated in the instrument developed in the present study, including a change to the 
response scale used. The original UTAUT questionnaire used 5-point Likert scales. In the 
present study, the items from the UTAUT were converted to bipolar statements. Chin et al. 
(2008) referred to bipolar scales as essentially semantic differential scales in which the rating 
options are similar to Likert scales. The labels are devised as good/bad, weak/strong and 
efficient/inefficient (Osgood, 1952). Bipolar scales have been deployed in many studies on 


technology acceptance (Behrenbruch et al., 2013, Hsu & Kulviwat, 2006; Huang, 2005; 
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Leem & Sung, 2019; Pedersen & Nysveen, 2003; Toft et al., 2014). An example of a 


converted UTAUT item from Likert scale can be seen in Figure 17. 


— rn Disagree Neutral Agree Strongly 
1 2 3 + 5 
I would find the system useful m my job q q o dq q 
Bipolar Scale 
1 2 3 #4 =5 
I would not find the system useful in my noosg I would find the system useful m 


job my job 


Figure 17. Conversion of UTAUT Item to Bipolar Scale 


The argument for using a bipolar scale in this study is that the odd-numbered Likert scale 
may lead participants to confuse the mid-point response, ‘neutral’ with the response ‘N/A’ or 
‘do not know’. There have also been various concerns about the functioning of Likert-based 
item formats in the literature (Géb et al., 2007; Jamieson, 2004; Norman, 2010). Although 
bipolar items have been said to increase participants’ cognitive demands and random errors in 
measurement, these have often been found to function in a superior way compared to Likert 
scales (Chin et al., 2008; Friborg et al., 2006; Hawkins et al., 1974; Van Auken & Barry, 
1995; Verhagen et al., 2015; Wirtz & Lee, 2003). While there was some initial concern over 
the increased number of words required for a bipolar scale, given that this instrument is 


designed for use with adults, this was deemed ultimately not to be a significant problem. 
Validation of the Instrument 


According to Messick (1988), the validity of an instrument is best examined through an 

overall evaluative judgement of the measure. Such a judgement is based on empirical 

evidence and theoretical rationales that support the adequacy and appropriateness of 

inferences and actions made. Messick (1995) proposed five criteria upon which new and 

existing instruments could be evaluated in terms of validity: 

(i) | Evidence based on test content, i.e., whether the content of the test is aligned with the 
intended focus of the instrument, which is commonly assessed by obtaining feedback 


from individuals who can reasonably make such judgements; 
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(11) Evidence based on response processes, i.e., whether what the respondents need to do in 
order to answer the questions is aligned with the intended focus of the instrument, 
which is commonly assessed using a ‘cognitive interview’ with representative 
participants; 

(111) Evidence based on internal structure, i.e., whether the factor structure and internal 
consistencies for the instrument support the proposed theoretical structure; 

(iv) Evidence based on relationships with other variables, i.e., whether the instrument scores 
correlate in predicted ways with scores from instruments that are theoretically related to 
the target construct/s; 

(v) Evidence based on the consequences of usage, i.e., whether the uptake or adoption of 


the instrument has a positive washback effect on practices. 


The validation of the instrument using the survey data was conducted to address three of the 
five criteria stipulated by Messick (1995). Specifically, this validation considered the 
instrument properties in terms of evidence based on the instrument’s content, associated 
response processes, and internal structure. The validation did not explore evidence based on 
relationships with other variables, as this was beyond the scope of the data collection possible 
within the institution. The other criterion that was not addressed was evidence based on the 
consequences of testing, which cannot be evaluated until the instrument has been used widely 


over a period of time. 


In line with Messick’s (1995) criteria, for the development and validation of the instrument to 
assess the acceptance of online assessment based on the extended UTAUT model proposed, a 
three-stage approach was used. Stage | involved the evaluation of the constructs and items 
through a content validity study. Stage 2 involved a cognitive interview with selected 
participants from the ITE. Following these two stages, a revision of the instrument items was 
conducted, and a final version of the questionnaire items generated. The final stage, Stage 3, 


involved a large sample of educators to assess the internal structure of the instrument. 
Participants and Procedures 
Prior to conducting the research, ethics approval was obtained from the researchers’ 


institution, and approval was also obtained from the ITE board. Stage 1 commenced with the 


evaluation of the constructs and items through the conduct of a content validity study. 
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Content validity is a product of the degree to which items in an instrument are relevant to, 
and representative of, their intended constructs (Haynes et al., 1995; Lawshe, 1975; Messick 
1987). Tojib and Sugianto (2006) presented a literature review on content validity in 
information system research studies and found that qualitative approaches to investigating 


content validity were preferred. 


A qualitative pilot study was therefore used here with a group of 7 lecturers to explore the 
initial 34-item questionnaire. The participants were all IAS users from ITE who were 
involved in the initial pilot implementation phase and had delivered at least an online test and 
examination prior to this study. The participants each reviewed the questionnaire items for 
readability, clarity and comprehensiveness during individual interviews. They also suggested 
whether the items should be included in the final questionnaire, based on their judgments on 
their relevance, and representativeness. The content validity study during the pilot ensured a 
similar understanding of the item wordings between the researchers and the respondents. 
Participants were also asked to provide recommendations on ways to improve the sentence 
structures to ensure clarity and conciseness based on any difficulties encountered in 


comprehending the questionnaire instructions, questions and statements. 


In the pilot study, the amount of time required to complete the questionnaire was also 
estimated. Given that this indicated that a significant investment of time would be required to 
respond to all 34 items, the set was ultimately reduced to a collection of 20 items after 
reviews by the researchers. Items were chosen for retention by the researchers based on the 


perceived clarity of the items used to represent each of the model constructs. 


Stage 2 involved a cognitive interview (Presser et al., 2004) with three selected participants 
from the ITE. This was conducted to examine the thought processes that would be involved 
in interpreting the questions and arriving at responses for the questionnaire items. The 
purpose of the cognitive interview was to inform item revision decisions, and to provide 
further evidence on the clarity and relevance of item and response processes (Castillo-Diaz & 
Padilla, 2013). The staff members from the ITE headquarters were recruited to participate in 
the cognitive interview, who were heavily involved in the system development during user 
acceptance testing and like the first author, were administrators of the online assessment 


system. 
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The participants were chosen as they were familiar with the online assessment system and 
were deemed not in any conflicting power relationship with the researcher. Using verbal 
probes, the interviewees described their thinking after they had completed the questionnaire 
items. The objective was to identify items in which there was a misalignment between the 
interviewees’ interpretations and the intended meanings of the items, and to identify ways to 
modify the items based on their responses. Moreover, this stage provided further evidence 
related to content validity, as the participants provided feedback on the adequacy of content 


coverage. 


The final stage, Stage 3, involved conducting a model fit test using a large sample of 
educators. For the large-scale validation of the instrument in terms of internal structure, an 
email invitation was sent out to 469 potential participants, and 287 responded. From this 
group, 213 participants provided complete responses (1.e., overall response rate 45.42%). The 
objectives of the study were shared in the invitation email and before the start of the online 


questionnaire. 


Among the lecturers who responded to the online questionnaire, 62.0% were males, and 
38.0% were females. Most of the participants were from the age group of 41 to 50 years 
(39.4%), with a small group of participants from the 21-30 year age group (1.4%). For online 
assessment experience, 77.5% of the participants rated themselves at least “5” and above, and 
0.9% of the participants responded that they did not have any experience. Participation was 


voluntary and anonymous. The profile of the participants is shown in Tables 3, 4 and 5. 


Table 3. Profile of Participants (within Age Groups) 


Age 
21to30 31to40 41to50 51to60 Above60 _ Total 


years old years old yearsold yearsold years old 


Gender Male Count 2 23 55 31 19 132 
Percentage 66.7% 47.2% 65.5% 72.1% 63.3% 62.0% 
Female Count 1 28 29 12 11 81 
Percentage 33.3% 52.8% 34.5% 27.9% 36.7% 38.0% 
Total Count 3 53 84 43 30 213 


Percentage 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 
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Table 4. Profile of Participants (within Gender Groups) 


Age 
21to30 31to40 41to50 51to60 Above 60 Total 
years old yearsold yearsold yearsold years old 


Gender Male Count 2 25 55 31 19 132 
Percentage 1.5% 18.9% 41.7% 23.5% 14.4% 100.0% 
Female Count 1 28 29 12 11 81 
Percentage 1.2% 34.6% 35.8% 14.8% 13.6% 100.0% 
Total Count 3 53 84 43 30 213 


Percentage 1.4% 24.9% 39.4% 20.2% 14.1% 100.0% 


Table 5. Participants’ Online Assessment Experience 


Online Assessment Experience 


i 2 3 4 5 6 fi Total 

Gender Male Count 2 1 3 23 39 46 16 132 
Percentage 1.5% 0.8% 2.3% 18.9% 29.5% 34.8% 12.1% 100.0% 

Female Count 0 2 1 14 31 19 14 81 
Percentage 0.0% 2.5% 1.2% 17.3% 38.3% 23.5% 17.3% 100.0% 

Total Count 2 3 4 39 70 65 30 213 


Percentage 0.9% 14% 1.9% 18.3% 32.9% 30.5% 14.1% 100.0% 


Results and Discussion 


A qualitative analysis was conducted for content validity and cognitive interviews, while the 
internal structure was assessed using quantitative education measurement analyses. During 
the content validity and cognitive interview stages, short notes stating the participants’ 
general impressions and immediate reflections were taken after each session. The feedback, 
comments and recommendations provided by the participants were considered when the 
questionnaire items were reviewed. The internal structure of the instrument was then 


evaluated using the statistical software, IBM SPSS AMOS version 26.0. 

Content Validity 

As indicated previously, participants in the content validity study indicated that each of the 
items in the questionnaire represented its intended construct well. Some feedback was given 


on how to clarify the meaning of specific items, which were modified or removed in response 


to these comments. For example, the participants pointed out that they did not understand the 
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phrase, “people who influence my behaviour’. The statement was thus modified in response 
to these comments to, “My supervisor thinks that I should use IAS”, “My colleagues think 
that I should use IAS” and “My students think that I should use IAS”. Overall, however, it 
was clear from this study that a 34-item questionnaire would be somewhat time-consuming to 


complete. As a result, a reduced, 20-item version was adopted later in the study. 
Cognitive Interviews 


The cognitive interviews identified several statements that were misunderstood by the 
interviewees. Specifically, the findings from the cognitive interview sessions resulted in the 
identification of four issues in the questionnaire items: (1) some statements were seen to be 
irrelevant in the way that they were worded; (2) some items were very similar and perceived 
to be repetitive; (3) some statements could be more extreme, being on the bipolar rating scale; 
and (4) some participants were unable to understand the words used in some statements. 


Based on the comments and feedback given, several items were either modified or removed. 
Internal Structure 


The model fit of the extended UTAUT was examined through a confirmatory factor analysis 
(CFA), conducted using IBM SPSS AMOS version 26.0. The CFA was conducted to 
estimate whether the actual factor structure and factor loadings aligned with the theorised 
structure, by statistically testing the fit between the observed correlations and the proposed 
measurement model (Albright & Park, 2009; Bollen, 1989; Hair et al., 2006; Kline, 2005). 
Descriptive statistics and bivariate correlations for the items in the instrument are shown in 


Table 6. 


Five indices were used to assess the fit of the model to the data: (a) y’/ Degree of Freedom 
(y°*/df), (b) Root Mean Square Error of Approximation (RMSEA; Steiger, 1990), (c) 
Goodness-of-Fit (GFI), (d) Comparative Fit Index (CFI; Bentler, 1990) and (e) Tucker-Lewis 
fit index (TLI; Bentler & Bonett, 1980). Because ° is overly sensitive to the sample size (Hu 
& Bentler, 1999), the ratio of y’ to its degrees of freedom (y/df) was used, and a range of not 


more than 3.0 was indicative of an acceptable fit (Kline 2005). 
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For the RMSEA, values of less than .05 are deemed to indicate a close model fit; those 
between .05 and .08 a reasonable fit; those between .08 and .10 a mediocre fit; and those 
greater than .10 an unacceptable fit (Browne & Cudeck, 1993). According to Hair et al. 
(2010), the value of GFI should be more than .95. The CFI and TLI compare the 
hypothesised model to a ‘null’ or worst-fitting model, taking into account model complexity, 
and indicate an acceptable model fit when values are greater than .90, while a good model fit 
is indicated when values are greater than .95 (Hu & Bentler, 1999). Three CFAs were 
conducted to identify the best model in this study. Table 7 shows the fit indices for the three 
models tested. Model 1 was a single-factor model, with all items in the instrument loading on 
a single factor. Model 2 is the extended UTAUT proposed with usability, learnability, PE, 
FC, SI, BI, attitude and UB as separate constructs. Model 3 was an additional analysis in 
which the two new constructs (usability and learnability) were combined into a single factor 
(‘adoptability’). This latter model was tested to determine whether both constructs should be 


retained separately, or could be considered elements of a single predictor. 


Table 7. Goodness-of-fit Indices for Extended UTAUT 


Values / Fit 
Measure Thresholds Model 1: Model 2: Model 3: 
1- Factor 8-Factor 7-Factor 

x? -- 1353.042 242.998 334.875 

df -- 170 142 149 
<3 good; 

lat <5 ie ee permissible aa phe ate 

p-value < 0.05 0.000 0.000 0.000 
< 0.05 (good); 

RMSEA 0.05 to 0.10 (moderate); 0.181 0.058 0.079 
> 0.10 (bad) 

GFI > 0.95 0.582 0.899 0.862 
> 0.95 (great); 

CFI > 0.90 (traditional); 0.729 0.977 0.955 
> 0.80 (sometimes permissible) 

TLI > 0.9 0.697 0.969 0.943 


The change in ’ across the three models was then used to evaluate whether the fit of the 
three models differed significantly. From the indices, this analysis indicated that the changes 
in x° were all significant. Specifically, the 7-factor Model 3 fit the data significantly better 
than the one-factor (Model 1), Ay* (21) = 1018.167; p<.01. However, the original 8-factor 
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model (Model 2) produced a significantly better fit than the 7-factor, in which the two new 
constructs were combined (Model 3), Ay? (7) = 91.877; p<.01. Given these results, the 
originally proposed 8-factor Model 2 was retained. The values for Model 2 fell within the 
recommended thresholds for acceptable model fit based on all other indices (y7/df = 1.711; 
RMSEA = .058; GFI = .899; CFI = .97, TLI = .969). Both the (x? df and RMSEA were low 
for Model 2 and within recommended levels, while the GFI, CFI and TLI were all higher than 
the recommended cutoffs. Figure 18 presents the path coefficients for Model 2. An internal 
item consistency test was conducted for the overall model with a Cronbach's Alpha 
coefficient attaining .960 (Cronbach, 1951). The internal item consistency for all individual 


constructs is also calculated, ranging from .765 to .975 (see Table 8). 


Table 8. Cronbach's Alpha Coefficients 


Construct Item Cronbach's Alpha 
Usability ASUU_1 
ASUU_2 0.901 
ASUU_3 
Learnability ASUL_ 1 
ASUL 2 
PE BCRPE 2 
BCRPE 3 
SI BCRSL 1 
BCRSI 2 0.908 
BCRSI 5 
FC BCRFC_1 
BCRFC_ 3 0.839 
BCRFC_4 
BI CARBI 1 
CARBI 3 
Attitude CARA 1 
CARA 2 0.975 
CARA 4 
UB DBRUB. 1 
DBRUB 3 


0.801 


0.953 


0.948 


0.765 


Conclusion 


The primary objective of this study was to validate a new instrument designed to measure 


users’ acceptance of online assessment. The instrument was based on an extension of the 
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original UTAUT model. Previously, there has been a limited amount of research that has 
adopted the UTAUT to study online assessment adoption. The study by Tan (2013) on 
students’ adoptions and attitudes towards computerised placement tests was the one 
exception identified by the authors. In other studies, the examination of responses to online 
assessment has always been linked to responses to online learning systems in general (Hillier 
et al., 2018), which can confound the acceptance of learning and assessment elements. 
Further studies that focus specifically on users’ acceptance of online assessment systems 
would enable institutions to better adapt the change management practices they adopt in 
relation to technology-based systems, to ensure that this element of an overall online 


approach is received favourably. 
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Figure 18. Confirmatory Factor Analysis of Extended Unified Theory of Acceptance and Use 
of Technology 
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The instrument developed and proposed in this study demonstrated sound psychometric 
properties based on evidence of content, response process, and internal structure validity. 
Thus, with minor revisions to the item wordings (e.g., replacing reference to the IAS with the 
name of the local system under study), this instrument holds significant promise for use in 
future studies that focus on lecturers’ acceptance of online assessment approaches in higher 
education. Given the somewhat disparate findings reported in previous UTAUT studies, 
however, further research into its psychometric properties would be needed to ensure that the 


instrument is suitable for use across a wide range of participants and settings. 
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CHAPTER 5: DETERMINANTS OF ONLINE ASSESSMENT 


ADOPTION IN A SINGAPORE TECHNICAL COLLEGE 


Pen-and-paper examinations have remained the traditional approach to student assessment in 
education institutions over the last few decades. Despite these enduring habits, the 
widespread availability of learning management systems and technology has increased the 
popularity of online assessment approaches in more recent years (Amasha et al., 2018; Choi 
& McClenen, 2020; Gamage et al., 2019; Krusche & Seitz, 2018; Liu et al., 2019; Xu & 
Mahenthiran, 2016; Way et al., 2020). Online assessment has the potential to enhance the 
process of evaluating student learning in myriad ways. For example, this approach can: 
increase testing reliability with machine marking; improve impartiality in assessment; permit 
the use of diverse question styles that incorporate interactivity and multimedia (Boyle & 
Hutchison, 2009; James et al., 2002); and increase lecturers’ ability to test a wide variety of 


topics within a single test, in a short time period (Brady, 2005). 


With features like automated marking and feedback, online assessment is viewed as efficient, 
fast and reliable, making it useful where large numbers of students are being tested. Online 
assessment may not only reduce the instructional and administrative costs of teaching a large 
class, but indirectly, affect the amount of student learning that takes place in courses by 
lowering the costs associated with administering more frequent assessments. Online 
assessment has the potential to support and even improve student learning with properly 
designed assessment tasks. Moreover, higher-order assessment tasks can also be assessed 


through online approaches. 


Studies have shown that in general, students respond well to online assessment approaches, 
particularly if the assessment results are counted towards their final marks (Appiah & Van 
Tonder, 2018). However, it is the lecturers who must first prepare and then deliver these 
online assessment tasks. This group, therefore, will have a key influence on how successfully 


an online assessment system is used within an institution. While numerous studies on 
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lecturers’ acceptance of online /earning systems have appeared within the literature, studies 
that have focused on their acceptance of online assessment systems are relatively scarce. The 
present study focused on exploring the factors that predict lecturers’ acceptance of one 
specific online assessment system used within an institute of technical education in 
Singapore. The central model used to explore lecturers’ acceptance of this system was an 
extended version of the Unified Theory of Acceptance and Use of Technology proposed by 
Venkatesh et al. (2003). 


Venkatesh et al. (2003) consolidated various previous Technology Acceptance Model (TAM) 
theories (Davis, 1989; Taylor & Todd, 1995) and models (Ajzen, 1991; Compeau et al., 
1999; Fishbein & Ajzen, 1975; Moore & Benbasat, 1991; Thompson et al., 1991) in 
developing the Unified Theory of Acceptance and Use of Technology (UTAUT). In the 
UTAUT, four constructs play a significant role as direct determinants of behavioural 
intentions (BI) and use behaviours (UB) with respect to any technology-based system: (1) 
performance expectancy (PE), (2) effort expectancy (EE), (3) social influence (SI); and (4) 
facilitating conditions (FC) (see Figure 19). 


Performance Behavioural 
Expectancy Intention 
Effort 
Expectancy 
Social 
Influence 
Facilitating 
Conditions 


Figure 19. Unified Theory of Acceptance and Use of Technology Model [Venkatesh, V., 
Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information 


Use 
Behaviour 


technology: Toward a unified view. MIS quarterly, 425-478. ] 


PE is the degree to which a user believes that using the system will help him or her to attain 
gains in his or her job performance, and has been found to be a determinant of behavioural 
intentions in most situations. EE is the degree of ease with which the system can be used. SI 


is the extent to which a user perceives that ‘important others’ think that he or she should use 
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the system. FC is the extent to which a user believes that there are existing organisational and 
technical structures to support their system usage. Finally, BI is defined as the individual’s 


intention to use the technology, while UB is their actual usage behaviour. 


While the UTAUT has been found to have a high level of predictive power in explaining 
users’ acceptance of various information systems (Venkatesh, et al., 2003), the original 
developers acknowledged that the model could be extended for use in specific contexts. As a 


result, an extended UTAUT model has been proposed by the first author as depicted in Figure 


20. 
ie 


Behavioural 
ee ; Intention 


Facilitating 
Conditions 


Figure 20. Extended Unified Theory of Acceptance and Use of Technology Model 
[ Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of 
information technology: Toward a unified view. MIS quarterly, 425-478. ] 


As indicated, in the extended model, PE, SI and FC have all been retained, because these 
have been shown to be powerful predictors of technology acceptance in previous research 
(Venkatesh, et al., 2003). However, two additional constructs (usability and learnability) have 
been added as potential influences on BI, given that these factors have been identified as 
important predictors in studies outside the UTAUT literature (Burney et al., 2017; Chiou et 
al., 2009; Holden & Rada, 2011; Jeng, 2005; Juarez Collazo et al., 2014; Lah et al., 2020; 
Lin, 2013; Tsakonas & Papatheodorou, 2008; Zbick et al., 2015). The construct of attitude 
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has also been added, as this factor has been found in various previous studies, including 
UTAUT extension studies, to be a significant predictor of both BI and UB (Botero et al., 
2018; El-Gayar and Moran, 2006; El-Gayar et al., 2011; Jairak et al., 2009; Khechine, & 
Augier, 2019; Moran et al., 2010; Nassuora, 2012; Shuhaiber, 2015; Thomas et al., 2013). 


In the extended model, usability is defined as the ease with which a system can be adopted to 
achieve given objectives with effectiveness and efficiency (Bevan et al., 2015; Jokela et al., 
2003; Shackel, 2009). Learnability refers to the speed with which users can become familiar 
with the features and capabilities of a system, and will depend heavily on the quality of the 
system interfaces (Jeng, 2005; Nielsen, 1994). Both usability and learnability have yet to be 
evaluated as constructs in a UTAUT study, though these variables have been explored in past 
technology acceptance research. For example, Lin (2003) found that learnability (as an 
element of usability) was a significant predictor of intentions to use an eCampus learning 
system with a personal digital assistant (PDA), while Zbick et al. (2015) found that 
learnability was a significant predictor of intentions to use a mobile learning system in a 


university-level setting. 


Attitudes (i.e., users’ overall subjective thoughts or feelings about an object), which first 
appeared in the earlier TAM model but was not incorporated as a construct in the final 
UTAUT model, been re-introduced in the extended version proposed. According to the 
Theory of Reason Action, individuals’ attitudes towards a given object or situation combine 
with subjective norms to shape their behavioural intentions, which also influence their actual 
behaviours (Ajzen & Fishbein, 1980). In the UTAUT extension studies on tablet PC adoption 
by El-Gayar and Moran (2006), Moran et al. (2010) and El-Gayar et al. (2011), attitudes 
toward using technology were found to significantly influence BI. Attitudes have also been 
found to influence BI in studies of mobile learning adoption (Nassuora, 2012; Thomas et al., 
2013), virtual lecturing systems, mobile-assisted language learning systems and social 


learning platforms (Botero et al., 2018; Khechine & Augier, 2019; Shuhaiber, 2015). 


In the exended UTAUT model, user acceptance is still operationalised by two of the 
constructs from the original UTAUT (BI and UB). Both of these have been incorporated in 
the model, though many later UTAUT researchers only considered BI as an outcome variable 
in technology acceptance studies. For example, Wong et al. (2013) did not include UB when 


studying student teachers' acceptance of interactive whiteboards using the original UTAUT 
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model. In another extended UTAUT study, Bouznif (2017) excluded UB in the exploring 
business students' continued intentions to use a learning management system. Although 
Dwivedi et al. (2019) argued, based on various UTAUT studies, that the influence of BI on 
UB might not be particularly strong or predictable, users’ actual behaviours (whether 
assessed directly from records such as computer logs, or through a self-report survey) are an 
important indicator of overall acceptance levels. UB is, therefore, included as an endogenous 


variable in the extended UTAUT model. 


While we recognised that in the original UTAUT model, the influence of the four primary 
factors (i.e. PE, EE, SI and FC) on BI and UB might be moderated by factors such as gender, 
age, experience and voluntariness, these factors are not typically included in empirical studies 
on the UTAUT. As a result, the extended model evaluated here does not include reference to 
these potential moderating variables. Further empirical studies that adopt the extended model 


could, however, include these as additional factors. 


The goal of the present study was to explore lecturers’ acceptance of one online assessment 
system (the Integrated Assessment System, or IAS) in the context of an institute of technical 
education (ITE) in Singapore. In the study, 213 lecturers compeleted an online survey based 
on the extended UTAUT model. A path analysis was then conducted to examine relationships 
between the constructs in the extended model. The following hypotheses regarding 
relationships between the constructs in the extended UTAUT model were tested in the study: 

e Usability has a significant positive effect on attitude, BI and UB 

e Learnability has a significant positive effect on attitude, BI and UB 

e PE has a significant positive effect on attitude, BI and UB 

e Sl has a significant positive effect on attitude, BI and UB 

e FC has a significant positive effect on attitude, BI and UB 

e Attitude has a significant positive effect on BI 


e Bl has a significant positive effect on UB 


The research model and its hypotheses are also shown in Figure 21. It should be noted here 
that there were no explicit hypotheses formed for indirect effects within the model, because 
previous studies have not focused upon these. As a result, there was no basis on which to 


form such predictions. However, these also were tested and interpreted in the study. 
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Furthermore, it should be noted that as effort expectancy was not included in the instrument, 
it was not possible to do a direct comparison of the strength of prediction from the extended 


model and the original UTAUT. As a result, this difference was not tested in the research. 


Facilitating 
Conditions 


Figure 21. Hypotheses of Extended UTAUT Research Model 


Method 


Participants 


An online questionnaire was used to obtain information from participants who had used the 
IAS at least on one occasion. From the email invitations sent out to 469 potential participants, 
213 participants completed the online questionnaire. The online questionnaire response rate 
was 45.42%. Among the participants who responded to the online questionnaire, 62.0% were 
males, and 38.0% were females. Most of the participants were from the age group of 41 to 50 
years (39.4%), with a small group of participants from the age group of 21-30 years (1.4%). 
In terms of online assessment experience, 77.5% of the participants rated themselves at least 


“5” and above, and 0.9% of the participants responded that they did not have any experience. 
Instrument 


The instrument used in the study included 20 items and utilized a 7-point bipolar rating scale 


(see Table 9). Bipolar rating scales have been found to reduce bias and produce better model 
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fits in instrument analyses (Friborg et al., 2006). In the instrument, each bipolar item contains 


two full statements to ensure meaningful responses and avoid ambiguity. As indicated, in this 


study, it was not possible to study actual usage behaviours, because permission to access 


these data could not be obtained. As a result, the measure of UB relied upon participants’ 


self-reports of of the frequency with which they used the IAS. 


Table 9. Extended UTAUT Questionnaire (20 Items) 


Construct Statement 
Performance Using IAS slows me down in ex Using IAS enables me to 
Expectancy (PE) accomplishing tasks accomplish tasks quickly 
(Venkatesh et al., Using IAS decreases my es Using IAS increases my 
2003) productivity terribly productivity greatly 
My supervisor thinks that I ene My superior thinks that I 
should not use IAS should use IAS 
Social Influence 
My colleagues think that I My colleagues think that I 
(SI) €> 
should not use IAS should use IAS 
(Venkatesh et al., — — 
2003) In general, the organisation has In general, the organisation has 
been unsupportive in the use of ©? been supportive in the use of 
IAS IAS 
I do not have any resource es I have all the resources 
ae available to use IAS available to use IAS 
Facilitating _ ; ; : ; ; 
. IAS is incompatible with IAS is compatible with all other 
Conditions (FC) <2 
systems I use systems I use 
(Venkatesh et al., ; ; 
2003) No specific person (or group) A specific person (or group) is 
is available for assistance with ©? _ easily and readily available for 
IAS difficulties assistance with IAS difficulties 
I think IAS is extremely es I think IAS is extremely easy to 
difficult to use use 
Usability I think the various functions in es I think the various functions in 
(Brooke, 1996) IAS are not integrated IAS are well-integrated 
I think there are too many es I think there is overall 
inconsistencies in IAS consistency in IAS 
€> 


Learnability 


I need the support of a 


I am able to use IAS on my 
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Construct Statement 
(Brooke, 1996) technical person to be able to own without any help 
use [AS 
I need to learn a lot of things I need not learn new things 


before I could get going with <> before I could get going with 


IAS IAS 

Behaviourial Given a choice, in the next 6 months: 
Intention (BI) I do not intend to use IAS <2 I intend to always use IAS 
(Venkatesh et al., 

I do not plan to use IAS bcd I plan to always use IAS 
2003) 

All things considered, using IAS is: 
Attitude Bad 2 Good 
(Davis, 1989) Unfavourable ©? Favourable 
Negative c? Positive 

Use Behaviour I seldom use IAS “ I always use IAS 
(UB) (Venkatesh I spend great amount of time on 

I spend little time on IAS =? 
et al., 2003) IAS 

Procedures 


Following the receipt of approval both from the participating institution and the University of 
Western Australia Human Research Ethics Committee, the questionnaire was completed 
online, hosted on the Qualtrics platform. Given the busy schedules of the lecturers, it was left 
open for three months to ensure that all participants had the opportunity to complete the 
survey. The objectives of the study were shared in an invitation email, and on the landing 


page of the online questionnaire. Participation was voluntary and anonymous. 

Results 

Descriptive statistics and bivariate correlations for the path analysis are shown in Table 10. 
All initial screening analyses performed suggested that the use of path analysis was tenable, 


indicating no significant violations of assumptions in terms of non-normality, non-linearity, 


or extreme scores. 


ot 


Determinants of Online Assessment Adoption in Singapore Technical College 


Table 10. Descriptive Statistics and Bivariate Correlations 


Construct M SD 12 3 4 5 6 7 8 
1. Usability 4.814 1.104 -- .515** .733** .653** .650** .669** .723** 468** 
2. Learnability 4.146 1.340 -- AS58** SOS** SLL** SI5** 440** 465** 
3. Performance 

4.880 1.190 =e .664** 630** 676** .752** 464** 
Expactancy 
4. Facilitating 

5.452 1.088 = 688** .777** .731** 649%* 
Conditions 
5. Social Influence 4.967 1.048 — .701** .680** .602** 
6. Attitude 5.143 1.186 = 828** 684** 
7. Behavioural 

5.355 1.147 -- 593** 


Intentions 


8. Usage Behaviours 4.862 1.324 a 


* Significant at .05 level; ** Significant at .01 level 


The path analysis using IBM SPSS AMOS version 26.0 indicated that usability had a highly 
significant positive effect on attitude (b= .294; p< .00/) (see Figure 22), while Learnability 
had a significant positive effect on BI (6= .138; p< .05) and UB (8 = .163; p< .05). 


— 


** n<0.001 
* p<0.05 


aii a p>0.05 


Vv 
Facilitating 
Conditions 


Figure 22. Extended Unified Theory of Acceptance and Use of Technology Structural Model 


PE had a significant positive effect on attitude (B= .476; p< .001) but did not have a 
significant effect on BI, a departure from common UTAUT findings that have suggested that 
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PE is the strongest predictor of BI. SI was found to have positive effect on attitude (B= .39/; 
p< .001); BI (B= .389; p< .001) and UB (f= .362; p< .001). FC was found to have a 
significant positive effect on attitude (B= .232; p< .001), BI (B= .198; p< .001) and UB (f= 
.290; p< .001). Attitude emerged as the strongest predictor of BI (6= .523; p< .001). Similar 
to the original UTAUT findings, BI had a significant effect on UB (f= .393; p< .001). The 


test results for the variables are summarized in Table 11. 


Table 11. Path Coefficients of extended UTAUT Model 
Unstand. Stand. 


Hypothesis Path : ; S.E. CR. P Result 
Estimate Estimate 
Hl Usability Attitude .268 294 051 5.263 *** Significant 
Not 
H2 Usability> BI .000 .000 049.004 997 
Significant 
- Not 
H3 Usability >UB -.139 -.131 072 .053 
1.933 Significant 
Not 
H4 Learnability> Attitude  —-.031 -042 042 -.741 459 
Significant 
H5 Learnability> BI .104 138 039 2.653 .008 Significant 
H6 Learnability>UB 141 .163 061 2.297 .022 Significant 
H7 PE Attitude 348 476 040 8.705 *** Significant 
Not 
H8 PE>BI -.005 -006 044 -.109 .913 
Significant 
- Not 
H9 PE>UB -.099 -116  .058 091 
1.691 Significant 
H10 SI> Attitude 311 391 044 6.984 *** Significant 
H11 SIOBI 314 389 048 6.544 *** Significant 
H12 SI>UB 334 362 090 3.703 *** Significant 
H13 FC Attitude 192 232 047 4.062 *** Significant 
H14 FC SBI .166 198 046 3.612 *** Significant 
H15 FC>UB .279 .290 078 3.587 *** Significant 
H16 Attitude> BI 531 523 076 7.005 *** Significant 
H17 BI>UB 450 393 127 3.546 *** Significant 


In assessing the extent to which each independent variable has an impact on the dependent 


variables, the standardised direct effects, indirect effects, total indirect effects and total 
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effects associated with each of the five variables were examined. A coefficient linking one 
construct to another in the UTAUT model represents the direct effect of a determinant on a 
dependant variable. An indirect effect indicates the impact which a determinant has on a 
target variable through its effect on other intervening variables in the model. A total indirect 
effect on a given variable is the product of the indirect effects, while a total effect is the sum 
of the respective direct and indirect effects. According to Cohen (1988), effect sizes of 0.2 are 
considered small, those with 0.5 are medium, and values with 0.8 and above are considered 
large. These effects are summarized in Table 12. As indicated, Usability, PE, SI and FC all 
had significant indirect effects on BI, while SI and FC also had significant indirect effects on 
UB. These results indicate that, in addition to attitude having a significant direct effect on BI, 
it also acted as a significant mediator in the indirect effects of Usability, PE, SI and FC on BI 
(e.g., Usability > Attitude > BI). Attitude was also a significant mediator in the indirect 
effects of SI] and FC on UB, via BI (e.g., Usability > Attitude > BI > UB). 


Table 12. Direct, Indirect and Total Effects Implied in Path Model (standardized coefficients) 


Direct Effects Indirect Effect Total Effect 
Attitude BI UB Attitude’ BI UB Attitude’ BI UB 
Usability 294* 00 -.131 - 154* 061 .294* .154* = -.071 
Learnability -.042 .138* .163* - -022 046 =-.042 = .116* .209* 
PE 476* -.006 -.116 - 249* 095 .476* .242*  -.021 
SI ool .389" .362* - 204" 233" ~391* -.595* 595% 
FC saa" AOS" 290 - AZi* 4125" 232% 219° 416" 


Discussion 


There Attitude has appeared inconsistently in tested UTAUT models within the literature, 
more recently, being excluded by most UTAUT researchers in line with the original UTAUT 
model (Venkatesh et al., 2003). However, results of the present study suggest that lecturers’ 
attitude towards online assessment is an important predictor of both lecturers’ BI and UB, 
and in fact, was emerged as the strongest predictor of lecturers’ BI amongst all of the 
UTAUT predictors. This result is consistent with the findings of most of the previous studies 
in which attitude has been included (i.e., in a TAM or UTAUT extension study) (Bervell et 
al., 2020; Botero et al., 2018; Dulle & Minishi-Majanja, 2011; El-Gayar & Moran, 2006; EI- 
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Gayar et al., 2011; Khechine & Augier, 2019; Moran et al., 2010; Nassuora, 2012; Shuhaiber, 
2015; Sumak et al., 2010; Thomas et al., 2013). 


The path analysis also showed that there were significant indirect effects of the online 
assessment system usability, the lecturers’ PE and SI, and the organisation’s FC on lecturers’ 
BI through lecturers’ attitude. Such mediation analysis has only been a focus of one prior 
UTAUT study thus far. In this study, Bervell et al. (2020) explored the intended use of 
blended learning in an LMS application among 267 distance tutors in Africa and found 
significant indirect effects of PE and FC on BI, via attitude. Such a result is also consistent 
with the TAM model by Davis (1989). The former study cited also found that when attitude 
was added to the model, the direct effect of FC on BI was not significant, and that only an 
indirect of FC (via attitude) was significant. These previous results, coupled with those from 
the present study, underscore the important role that attitude can play in predicting both 
users’ intentions and their self-reported usage behaviours in relation to technology-based 


systems. 


The new construct, online assessment system usability, was theorised to have a significant 
positive direct effect on lecturers’ attitude, BI and UB. This prediction was not upheld, with 
results indicating no direct effects of this construct on either lecturers’ BI or UB. The fact that 
usability had no direct effects on users’ intentions was somewhat surprising but is consistent 
with results reported by Chiou et al. (2009) and Lew et al. (2019). Despite this, the results 
affirmed the importance of usability as a predictor of BI, because this construct had a 
significant positive indirect effect on BI, via attitude. Therefore, these results suggest that 
usability will first affect the lecturers’ attitudes towards online assessment, and through 
lecturers’ attitude, will have a significant impact on lecturers’ BI. The notion of attitude being 
an important mediator for behavioural intentions is not new (Bervell et al., 2020), but these 


constructs have not previously been incorporated in the context of the UTAUT model. 


In contrast to the findings for usability, learnability of the online assessment system did not 
have a significant direct effect on the lecturers’ attitude but did have significant positive 
direct effects on both lecturers’ BI and UB. This result aligns with past studies that have 
shown learnability to be an important factor in system usability assessments (Alshehri et al., 
2019; Thowfeek & Salam 2014). Learnability has previously been studied in the context of 


UTAUT extension studies, but only as an attribute within an overall usability construct (Lin 
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2003; Zbick et al., 2015). The findings of the present study, therefore, affirm the influence of 
learnability in technology acceptance. Furthermore, the disparate effects observed for 
usability and learnability in the study underscore the importance of decoupling these 


constructs in future UTAUT extension studies. 


The study findings also indicated that lecturers’ PE had a significant positive direct effect on 
the lecturers’ attitude towards online assessment, but not on BI and UB. The findings are 
similar to those of past studies that have examined the relationship between PE and attitude 
(Bervell et al., 2020; Botero et al., 2018; El-Gayar et al., 2011; Shuhaiber, 2015; Sumak et 
al., 2010), but are inconsistent with the predictions of the original UTAUT model. 
Traditionally, PE has always been proposed and found to be the strongest predictor of BI 
(Liao et al., 2004; Prasad et al., 2018; Salloum & Shaalan, 2018; Venkatesh et al., 2003), with 
a few notable exceptions (Mtebe & Raisamo, 2014; Yueh et al., 2015). It is possible that the 
findings of the present study were influenced by the fact that the use of the online assessment 
system was non-voluntary. As such, most of the lecturers were likely to deem that the system 
would help them to improve their job performance, because, in effect, they could not perform 


their functions fully without making use of it. 


From the findings, lecturers’ SI also had a significant positive direct effect on lecturers’ 
attitude, which aligns with results of past studies that have incorporated attitude as a construct 
(Botero et al., 2018; Nassuora, 2012; Shuhaiber, 2015; Sumak et al., 2010). SI among the 
lecturers had a significant positive direct effect on their intentions to use the online 
assessment system, which is also consistent with the findings of many previous UTAUT 
studies (e.g., Al-Adwan & Al-Adwan, 2018; Alasmari & Zhang, 2019; Ali & Arshad, 2018; 
Kim & Lee, 2020; Radovan & Kristl, 2017; Salloum & Shaalan, 2018; Shah et al., 2020; 
Wan et al., 2020; Zhang et al., 2020). In the present study, however, a new relationship 
between SI and UB was also identified, where SI had a significant positive direct effect on 
UB. The latter effect suggests a particularly strong influence of social influences in this 
context. Thus, within the participating ITE, the views of others (i.e., peers, supervisors, and 
students) are likely to have a significant impact in lecturers’ overall acceptance of any new 


online assessment system. 


The relationship between FC and attitude has seldom been explored in past UTAUT studies. 


In the only two studies identified in which this relationship has been explored, Nassuora 
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(2012) and Bervell et al. (2020) both found that FC had a positive influence on attitude. A 
similar result was obtained in the present study. The mediation analysis also revealed that 
there were significant indirect effects of SI and FC on UB, via BI. This result is consistent 
with the findings of one previous study, which focused on teachers' acceptance of 
communication technology by Shah et al. (2020). In this former study, it was also found that 
PE, SI and FC had significant indirect effects on UB through BI as a mediator. 


As has been reported in various prior UTAUT studies, FC in the present study had a 
significant positive direct effect on UB (Alshehri et al., 2019; Liao et al., 2004; Mahande & 
Malago, 2019; Oh & Yoon, 2014; Prasad et al., 2018; Salloum & Shaalan, 2018; Shah et al., 
2020). Typically, however, the relationship between FC and BI is not tested. In the present 
study, FC within participating ITE was also found to have a significant positive direct effect 
on the lecturers’ BI. In the two studies that have also tested this relationship to date, Mtebe 
and Raisamo (2014) found that FC had a positive effect on BI in a study of students' 
behavioural intentions to adopt and use mobile learning, while Mahande and Malago (2019) 
reported a similar effect in their study of e-learning acceptance in a postgraduate degree 
program. Therefore, the results of the present study have highlighted new relationships 
among the original and new constructs in the extended UTAUT model, which could provide 


useful directions for future research in which this model is adopted. 
Conclusion, Limitations and Future Directions 


Since its introduction, the UTAUT has been highly regarded as a robust model with a high 
level of predictive power in technology acceptance studies. Venkatesh et al. (2003), however, 
acknowledged that across different contexts, extensions to the original UTAUT could be 
considered. The present study added usability, learnability and attitude to the UTAUT model, 
and examined their relationships in predicting acceptance of a form of technology that has 


thus far been under-researched in the UTAUT literature (online assessment systems). 


The study results suggest that the new constructs introduced could enrich and expand 
explanations of the factors that influence users' intentions and usage in such settings. The new 
relationships amongst traditional elements of the UTAUT model also suggest a promising 
line for future studies to explore. The results suggested in particular that the introduction of 


attitude in extended models could enhance the efficacy with which user's intentions to engage 
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with online assessment systems can be predicted. This result also underscores the need for 
institutions to take steps to improve lecturers' attitudes towards the use of relevant technology 


systems in efforts to enhance their intentions and actual usage behaviours. 


While the results from this study confirmed many predictions that were a part of the original 
UTAUT model, and identified new relationships amongst the UTAUT constructs, it was not 
possible to incorporate potential moderators in the study. Such a study would require a very 
large and diverse sample. The examination of moderators could, therefore, be explored in 
future studies with respect to the new constructs introduced in this study. It was also, as 
noted, not possible within the study to obtain direct evidence of usage behaviours, which 
could be explored in future studies. These studies could also test the relevance of the 
extended UTAUT model in other contexts, for instance, examining its predictive power with 


respect to other types of technology. 


One other possible direction for future UTAUT research within education contexts is the 
study of links between organisational culture and technology acceptance, and how this factor 
contributes to influence lecturers’ intentions and use behaviours. The prominent role played 
by SI in this study suggests that this line could prove fruitful for enhancing overall 
acceptance levels. Furthermore, past studies have shown that there is a strong relationship 
between organisational culture and technological innovation (Huang & Teo, 2019; Zhu, 
2015). The addition of new variables like organisational culture to the existing UTAUT 
model could further enrich our understandings of how users respond to the introduction of 


new technological innovations within institutions of higher education. 
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CHAPTER 6: GENERAL CONCLUSIONS: 


DETERMINANTS OF ONLINE ASSESSMENT ADOPTION 


From the deployment of the first electronics assisted assessment in the 1960s to the online 
assessment systems that are currently available, interest in the use of online assessment 
processes has grown, particularly since the emergence of learning management systems (Baig 
et al. 2020; Gamage et al., 2019; Shdaifat & Obeidallah, 2019; Xu & Mahenthiran, 2016). 
For large-scale assessment implementations, online approaches are often coupled with 


commercial systems like Blackboard and Moodle (Jordan, 2013). 


The development of online assessment approaches is likely to continue to grow at increased 
rates as students become increasingly technology-savvy, and expect to take examinations and 
tests on computers and mobile devices. Online assessment systems are also becoming 
essential as the accountability demands upon universities increase, because these have the 
ability to amass useful learning data for analysis. The COVID-19 pandemic, which forced 
educational institutions around the world to continue teaching and learning in digital forms, 


again pushed the implementation of online assessment to the forefront (UNESCO, 2020). 


Despite its potential, for the benefits of online assessment to be realized within higher 
education insititutions, it must be accepted by the educators who set and oversee these 
assessments (1.e., the lecturers). The overarching goal of the research presented in this work 
was to contribute to addressing this issue in the context of one Singapore-based Institute of 
Technical Education. Previous research on lecturers’ acceptance of online assessment has 
appeared within the literature, but this research has not typically been based on a systematic 
and standardised theoretical model. The resulting lack of standardisation across studies makes 
comparisons across contexts, which can serve to highlight important moderating factors, 


difficult to make. 


In the present research, a well-established theoretical model on users’ acceptance of 
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technology was applied to examine the factors that predicted lecturers’ acceptance of the 
online assessment system used within the participating institution. This final chapter 
summarizes findings obtained in the research, and discusses implications for research and 
practice based on these findings. The limitations of the research reported, and directions for 


possible future studies based on the UTAUT model, are also discussed. 
Summary 


The overarching objective of this research was to identify the factors that predicted lecturers’ 
adoption of online assessment in one Singapore-based Institute of Technical Education. The 
factors investigated were system usability and learnability; lecturers’ performance 
expectancy, social influences and attitudes towards online assessment; and lecturers’ 
perceptions of the facilitating conditions that support online assessment. Among these 
factors, usability, learnability and attitude were introduced to the UTAUT model as additional 
variables. The adoption of online assessment was measured in terms of lecturers’ behavioural 
intentions to use the online assessment system and their self-reported system usage 


behaviour. The relationships between these factors were also examined. 


Four papers are presented in the book. The first reviewed previous literature on the Unified 
Theory of Acceptance and Use of Technology (UTAUT) model, and proposed an extended 
UTAUT model to examine the factors that influence online assessment adoption. The second 
reviewed studies in the area of online assessment methods published from 2007 to 2019. In 
the third paper, an instrument to assess constructs in the extended UTAUT model proposed 
was developed and validated. In the final paper, the instrument developed in the third study 
was used to explore relationships among the constructs in the extended UTAUT model. 
Specifically, path analysis was conducted to examine the various factors that influence 


lecturers’ acceptance of online assessment within the institution. 


Previous studies on online assessment have often been tied to studies on learning 
management systems (Hillier et al., 2018). Thus, questions of online assessment acceptance 
have typically been overshadowed by a focus on the acceptance of learning management 
systems in general (Schoonenboom, 2014). The focus of the present research on the online 
assessment aspect of such systems, therefore, is relatively rare amongst UTAUT studies. 


Furthermore, in many education-based UTAUT studies, the participants have often been 
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students. The present research also departed from this previous work by focusing specifically 
on lecturers, who will inevitably be key stakeholders in the introduction of any new 


technology-based system. 


The primary empirical contributions of this research were discussed in Paper 4, in which the 
constructs within the new extended UTAUT model were explored. This new model 
incorporated three constructs that were not included in the original UTAUT model: attitude, 
usability, and learnability. As discussed previously, the UTUAT model is an evolution from 
the TAM, which included attitudes as a predictor variable (Davis, 1986). During the 
development of UTAUT, however, the attitude construct was discarded because it was 
deemed to be redundant after taking into account the effects of the other model variables. 
Despite this, researchers have reintroduced attitude in as a construct in UTAUT extension 
studies, and it was likewise re-introduced in the extended model tested in the present 
research. The empirical results reported in Paper 4 make clear that attitude is an important 
predictor of users’ intentions and use behaviours. Thus, these results confirm that researchers 


should consider including attitude as a construct in future UTAUT extension models. 


The two other new constructs in the extended UTAUT model were also found significantly to 
predict, either directly or indirectly, users’ intentions and behaviours. While usability was not 
found to have a direct effect on intentions or reported behaviours, it did have a significant 
direct effect on attitudes, and a significant positive indirect effect on users’ intentions, via the 
attitude construct. Therefore, these results suggest that while usability does not directly affect 
users’ intentions and behaviours, it will ultimately affect these through its relationship with 
attitude. The notion of attitude being an important mediator for behavioural intentions is not 
new (Bervell et al., 2020). These constructs have not, however, previously been incorporated 


in the context of the UTAUT model. 


The other new construct included in the extended UTAUT model, learnability, behaved quite 
differently from usability in the study. Learnability did not have a significant direct effect on 
attitude, but instead, had significant positive direct effects on both intentions and reported use 
behaviours. This aligns with the results of past studies that have shown that learnability is an 
important factor in system usability assessments (Alshehri et al., 2019; Thowfeek & Salam 
2014). Collectively, therefore, the results reported in Paper 4 affirm the potential importance 


of both usability and learnability in predicting user acceptance levels. Their importance 


>? AMA 8s 


General Conclusions: Determinants of Online Assessment Adoption 


within the present study suggests that these factors would be worthy of consideration in 


subsequent UTAUT studies. 


The results reported in Paper 4 also confirmed those from previous studies, which have 
indicated that social influence factors have a significant direct effect on intentions, while both 
facilitating conditions and intentions have significant direct effects on actual use behaviours. 
However, in the present research, it was also found that social influences had a significant 
direct effect on usage behaviours, and that facilitating conditions had a significant direct 
effect on intentions. These relationships within the UTAUT have typically not been explored 
in previous studies. Thus, the findings of the present research identified new relationships 
that were not theorised in the original UTAUT model, and suggest, in line with the findings 
of Bervell and Umar (2017), that future researchers should explore a broader range of 


possible interconnections amongst the UTAUT constructs in their analyses. 
Implications for Practice 


From the research results, attitude played a key role in lecturer’s intentions to use the online 
assessment system within the institution. Specifically, attitude had the strongest direct effect 
on BI. Given this, within the participating institution and others, shaping the attitudes of 
lecturers towards new technology-based systems is important to ensure that these are used to 
their full potential. Several factors within the extended UTAUT model were shown, in turn, 
to be important in predicting lecturers’ attitudes towards the online assessment system. These 
were the perceived usability of the system, lecturers’ expecations that the system would 
enhance their performance, the social influences upon system use, and perceptions of the 
infrastructure available within the institute to support the use of the system. All of these are 
potentially alterable factors that could be manipulated to enhance lecturers’ attitudes and, in 
doing so, their intentions towards and actual usage of the system. For example, to enhance 
both PE and usability perceptions, institutions must ensure that the systems they adopt are ‘fit 
for purpose’ given the nature of the tasks for which they will be used within that institution. It 
implies that user needs analysis for such system must occur at the beginning of the 
development cycle, coupled with early significant user involvement, when it is being decided 


what the system will be like and how it is going to support and benefit the users. 


The research results showed that contextual factors like SI and FC, in addition to having a 
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direct effect on attitudes, also had positive and significant direct effects on both BI and UB. 
Learnability also had a direct effect on these two variables. Turning first to the findings for 
learnability and facilitating conditions, institutions must ensure that suitable support 
mechanisms such as helpdesks and training programs are available to staff when introducing 
such systems. Effective professional development should be provided to assist lecturers in the 
adoption of online assessment, which would not only enhance the conditions within which 
the system is used, but also, the perceived ‘learnability’ of the systems in the eyes of staff. 
Studies have shown that the most effective professional development programmes that 
improve teaching practices are those with activities that are ongoing and sustained over time 
(Tournaki et al., 2011). Besides providing professional development programmes, an ongoing 
institutional support structure is essential. Buchan and Swann (2007) suggested a three-level 
support approach, including real-time training and professional development, helpdesk 
troubleshooting and self-help resources. Educational institutions implementing online 
assessment systems should, therefore, considering a multi-faceted approach to professional 
development of staff, which incorporates varied training approaches and high-quality 
technical support structures for different faculty groups. These must be readily available and 
accessible throughout the assessment periods. The recommendations for the comprehensive 
training and support structure are also in line with the findings from earlier literature review 
on teachers’ acceptance where it was found that efforts to learn were required to use online 


assessment systems effectively (Amante et al., 2019; Rolim & Isaias; 2018). 


The relationship indicated between SI and UB was a new finding in this research. The 
influence of fellow lecturers, supervisors and students was significant, and this implies that 
maintaining a positive organisational culture in educational institutions is vital to technology 
adoption and usage. Educational institutions should strive to manage changes well and 
encourage “buy-ins’ from the lecturers to ensure that technological innovations can be 
accepted more readily and successfully. It is important also for all supervisors within 
institutions, who will no doubt be amongst the ‘important others’ whose expectations drive 
the intentions and behaviours of other staff members, to be openly supportive of any new 
systems adopted. Supervisors should play the role of sharing good practices and championing 
the use of online assessment systems within educational institutions. To further promote the 
use of online assessment, however, lecturers who have greater social and peer influence can 
also step up as “change agents” to lead the system adoptions and enhance the acceptance of 


these systems by others. 
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Directions for Future Research 


Based on the findings in this research, one of the critical directions for future UTAUT 
research is the study of the link between organisational culture and technology acceptance, 
and how it plays a part in influencing BI and UB. Studies have shown that there is a strong 
relationship between organisational culture features and technological innovation (Huang & 
Teo, 2019; Zhu, 2015). The addition of new variables like organisational culture to the 
existing model can further enrich current UTAUT extension studies and provide a more in- 
depth understanding of how educational institutions adopt and implement technological 


innovations. 


The extended UTAUT model and its associated instrument could be introduced in other 
institutes of higher learning in Singapore to compare the findings. In fact, there was only one 
previous UTAUT study that the author could locate which was conducted in the Singapore 
education context. This study focused on the adoption of wikis in a Singapore secondary 
school in 2013 (Toh, 2013). Further studies based on the UTAUT model could, for example, 
be applied to study Singapore students’ acceptance of online assessment approaches, which 


may identify different factors that promote acceptance to those identified for lecturers. 


This study collected and analyzed self-reported data to measure the lecturers’ UB, rather than 
analyzing user log files from the online assessment system. Self-reported data were used as it 
was not possible to obtain the actual system logs in this case (the institution in which the 
study was conducted was cautious about this point, owing to data privacy and security issues, 
particularly given that the system in question was used to deliver high-stakes examinations 
for national skills certifications). However, a future study could access the data analytics 
features of software systems to obtain a direct measure of users’ behaviour. This feature is 
currently undergoing development within the participating institution’s online assessment 


system. 


Like many prior UTAUT studies, this research also did not consider moderating variables to 
explain online assessment acceptance by the lecturers. It was not possible to look at 
moderating variables due to restrictions in the sample characteristics of this study. The 
examination of such moderators, such as in the study by Lin and Lai (2019), could be pursued 


in future studies, and in particular, with respect to the new constructs introduced in the 
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present study. For example, Venkatesh et al. (2003) argued that voluntariness had a 
moderating effect on users’ acceptance of new technology-based systems. Thus, further 
studies could explore the application of the extended model in other situations, in which 
system users have a greater degree of freedom in their adoption and usage decisions. Such 
research would provide a more in-depth and nuanced view of the influence of factors such as 
usability on acceptance, which would in turn permit more specific practical implications to be 


drawn for individual institutions. 


The current study took a quantitative approach to collect data from the participants, as the use 
of a survey instrument is the most efficient means by which a large amount of data can be 
collected. Despite this, future work could consider the utilization of qualitative or mixed 
methods designs (e.g., combining surveys, interviews and focus groups), to obtain a more in- 
depth understanding of the barriers to online assessment acceptance. Such an approach is 
likely to highlight a broader range of factors that affect users’ acceptance in different 
contexts. Armed with such knowledge, leaders within education institutions will be better 
placed to anticipate problems that may arise when introducing new online assessment 
systems, and thus, ensure that any associated change management processes are geared to 
address these problems. Such steps will, in turn, help to ensure that the full benefits of online 
assessment systems are reaped both within individual institutions, but also, within the higher 


education sector at large. 
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